The case of the missing data

Scientists don't always report everything they discover. Sometimes loose bits of data can be packed up in a box with a bunch of old books and research papers and left in a garage in Sydney, Australia. And that means intriguing, puzzling findings that didn't make sense back in the day, can slowly retreat into scientific oblivion, about to be lost to humankind for all eternity.

Except that, in this case, the phone rang and a determinedscientist from Bethesda, Md.having managed to track down one of the last surviving members of a research team, asked Boonseng Leelarthaepin if he knew what happened to the dataset from the Sydney Diet Heart Study that wrapped up in 1973. Luckily Leelarthaepin is a packrat and he knew where to look.

"I knew in my mind where it was" he told me. He went into his garage in Sydney and started moving piles of boxes. "It did take quite some time. I opened up all the boxes. There were a lot of old books and different things, like hard copies of published papers," he said. Buried deep in the pile, he found what Dr. Christopher Ramsden, back in Bethesda, was anxious to see. An obsolete 9-track magnetic computer tape from the original Sydney Diet Heart Study. It was an unremarkable looking brown spool with a faded 3M label that had the words "LEE 3" printed in blue ink.

"The recovery actually took a substantial amount of time because it was in a format that wasn't readable by today's standards," Dr. Ramsden told me. He's a researcher with the U.S. National Institutes of Health, and he knew there was some potentially important information on that magnetic tape.

Dr. Ramsden tracked down some old equipment, recruited some experts, and after much technical handwringing, and format configuring, the data, that had originally been stored on punch cards, was translated into modern computer language so that it could be reanalyzed and interpreted. And what Dr. Ramsden found in that data made headlines this week.

It was a second look at an old clinical trial, first published in the 1970s, that had set out to measure the dietary effects of saturated versus unsaturated fat. It was assumed that if blood cholesterol could be lowered by reducing saturated fat in the diet, lives would be saved. But in this study, the subjects who switched to unsaturated fat had a higher risk of death.

"That was really unexpected," Leelarthaepin told me on the phone from Sydney. Because unsaturated fat can lower cholesterol, "in theory, survival would be better," he said. "But it was the other way around."

Back then, Leelarthaepin was a research assistant on the study and, at the same time, he was collecting data for his own PhD on a different aspect of the research. I asked him what they were saying to each other in the labwhen they were faced with a puzzling increase in mortality. "We had no idea" he said. "We thought there must be some other factor influencing that." They adjusted the data, factoring in other risk factors like smoking, and still the observation held up. There seemed to be an increased risk of death from simply eating more vegetable oil.

"You control for age and all the other things, and that still did not explain it," Leelarthaepin said.

Almost half a century passed, and the Sydney Diet Heart Study languished in the medical literature, occasionally surfacing with a passing mention as a confounding example of the mysteries of dietary fat.

"Back in those days, all polyunsaturates were sort of considered the same," Dr. Ramsden said. "Mainly because they lowered blood cholesterol." Today, scientists know that polyunsaturated fat is much more complex. "In the last few decades there's been an increased recognition, and advancement in the science of this field," Dr. Ramsden said. "And it's now clear that there are multiple polyunsaturated fatty acids, and that each of them may have different biochemical and clinical effects."

Those polyunsaturated fats include omega-6 fatty acids and and omega-3 fatty acids. Most vegetable oils contain a combination of the two, but some oils are almost exclusively composed of omega-6s. Both are considered healthier than saturated fats, from animal sources.

So why did the Sydney Diet Heart Study report an increase risk of death in a group that ate a supposedly healthier oil? As an investigator in the emerging science of nutritional biochemistry, Dr. Christopher Ramsden wanted to know more. His first question: what kind of vegetable oil were the men in the study asked to eat?

"So, in that context, knowing there was an increased risk of death in the past, I really wanted to find out, number one, which oils were used."

Safflower oil affords rare glimpse

What he found out provided a unique research opportunity, because the original trial used safflower oil, which contains only omega-6 fatty acids, instead of a combination of omega-3s and 6s. An oil with a significant amount of omega-3s would complicate things, because omega-3shave been shown to have positive effects on the body, while omega-6s might cause inflammation and have other negative effects on the arteries. So by choosing safflower oil, with its concentrated levels of omega-6s, the Sydney Diet Heart Trial became one of the only clinical trials to investigate what happens if saturated fats are replaced with omega-6 polyunsaturated fatty acids.

"This trial, by the nature of the oil it provided, only increased omega-6 fats, specifically linoleic acid," Dr. Ramsden said. "And therefore it's really a rare opportunity."

In 1966, when they designed the trial, the Sydney researchers had no idea they were testing a future omega-6 hypothesis. They were just trying to find out what happened when middle aged men who'd already had cardiac events tried to replace saturated fat with unsaturated fat in their diet. What happened is that they did lower their blood cholesterol.Why then did they have increased mortality?The Sydney authors had no answer.

"Because we couldnt explain it," Leelarthaepin said, they reported the findings, and waited to "see if another study showed the same thing."

Forty years later, as Dr. Ramsden searched all of the scientific literature for research on polyunsaturates, the Sydney Diet Heart Study emerged as one of only three that were relevant to his meta-analysis. But there was a problem. While the other studies focused on death or illness from coronary and cardiovascular disease, for some reason, the Sydney authors didn't report the specific causes of death. Instead they the grouped the data into one large category: "all cause mortality."

Why didn't they analyze the mortality data in more detail? "The main reason is that we were running out of time, and had no funds," Leelarthaepin said. "We felt we had only one shot at it, and 'all cause mortality' was the one we chose." The other data was set aside. But now Dr.Ramsden wanted to see it.

"We recognized there was missing data from this trial that was stopping us from completing a thorough analysis," Dr. Ramsden said.

So that started the hunt through the boxes in Leelarthaepins garage, and ultimately led to the headlines this week: "Omega-6 Fats Linked to Increased Risk of Heart Disease." Because, when he updated the meta analysis, crunching the old Sydney numbers, Dr. Ramsden found there was "no evidence of cardiovascular benefit" when omega 6 was increased, "and a substantial signal for increased risk," he said.

There are limitations to the finding. Experts point out that the level of omega-6s in the Sydney Heart Diet Study was much higher than would be found in an average diet. But it was convincing enough for many to recommend taking a second look.

But there's another aspect to this study, one that has implications for the entire field of scientific research. And that is: what a difference missing data can make.

It prompted some finger wagging from the British Medical Journal, as an example of why all data from clinical trials should be made permanently accessible to the entire scientific community.

In a press release, BMJ editors said "We are asking researchers to tell us about any other documented examples of missing data, to build a picture of the full extent of the problem which is undermining evidence based medicine worldwide."

It's a crisis of lost information. Even though people volunteer to be test subjects, and scientists complete years of research, a large amount of clinical trial data never sees the light of day.

"The current best estimate is that half of all the clinical trials that are conducted and completed are never published," the BMJ press release said. And even when trials are published, without a chance to see all of the raw data, other scientists are not able to do an independent analysis and come to their own conclusions about what it all means.

"Missing clinical trial data may have an important impact on our overall ability to review the evidence base," Dr. Ramsden said.

Distorted scientific record

Two decades ago, the Journal of the American Medical Association published this warning: "Failure to publish an adequate account of a well-designed clinical trial is a form of scientific misconduct, that can lead those caring for patients to make inappropriate treatment decisions."

But so far efforts to recover missing data have failed. Back in 1997, dozens of medical journals offered an "amnesty," an open call for information about unpublished trials, so they could be tracked down and added to the international scientific record.

What happened? "Five years of industry resistance, government impotence and public confusion followed," Kamran Abbasi, acting editor of the BMJ, wrote in 2004.

"The failure to open up clinical trial data to independent scrutiny 'distorts the scientific record,' Abbasi wrote. "By suppressing negative findings and exaggerating positive ones, by downplaying harms and talking up benefits, healthcare decisions, are based on incomplete data and ultimately harm the patients."

So now 'open access' is being pursued once again. The Canadian Institutes of Health Research, and other public research funding agencies have put policies in place requiring open access to the data generated from studies they sponsor. But others warn that those policies do not apply to industry sponsored studies, "and cannot prevent biased publication and reporting of results from industry sponsored research."

There have been a series of well publicized examples of industry sponsored trials that failed to disclose potentially dangerous effects of the drug.Today, many scientific journals have a policy of compulsory registration, refusing to publish the results of any clinical trials that were not registered at the start of the study, with end points clearly defined, so everyone knows what the trial was setting out to investigate, and whether it worked, rather than choosing to highlight a secondary finding that was not part of the original inquiry.

But decades of research has already been lost, in the attics and basements and garages of scientists all over the world. Efforts to try to track it down have not always had the happy ending that Christopher Ramdsen was able to report. "And you know, we really think that's a testament to Leelarthaepin for his willingness to help us recover the data, go through it with us and help us evaluate it in detail," he said.

So how important is this missing data? In this case, the BMJ press office reported the relevance this way:

"The researchers conclude that recovery of these missing data 'has filled a critical gap in the published literature archive' and that these findings 'could have important implications for worldwide dietary advice to substitute omega-6 linoleic acid (or polyunsaturated fatty acids in general) for saturated fatty acids.'"

All of that because someone found an abandoned spool of magnetic tape sitting in a box in a Sydney garage.