The Great Unknown: Using Statistics to Explore the Secret Depths of Unpublished Research

I spent a large portion of the day today trying to figure out why a couple papers I have showed that EGCG, a component of green tea, increases glucose uptake into isolated skeletal muscle cells, but another shows the opposite.

The methods of these papers were a little different, and it's possible to speculate that some of the differences — for example, the amount of glucose in the culture dish — may have been responsible.

But it also occurred to me that perhaps EGCG doesn't have any effect on this phenomenon at all.  For all I know, perhaps a hundred times so far different groups have tested the effect of EGCG on uptake of glucose into skeletal muscle cells and 100 times it had no effect, and the research seemed like a waste of time and went down the memory hole.  But then a couple flukes occurred, and they were exciting, and the researchers wrote up the reports and got them published.

Melissa McEwen recently shared a New Yorker article with me, “The Truth Wears Off.”  It discusses the “delcine” effect — a scientist stumbles upon an exciting phenomenon, publishes about it, but then over time as others attempt to replicate it, or even as that very scientist tries to replicate it, the phenomenon seems to wear off, becoming much less true than it originally seemed.

Part of the decline effect is likely due to publication bias.  Negative findings just aren't very interesting and don't get published.  Later on, when everyone believes something to be true, it suddenly becomes interesting to publish contrary research.

The New Yorker article mentions John Ioannidis, a crusader against junk science who wrote a 2005 paper that had floated around the Native Nutrition list back in the day, entitled “Why Most Published Research Findings Are False.”  Ioannidis is a fascinating character, and you can read more about him in the Atlantic article, “Lies, Damned Lies, and Medical Science.”

Statistics is an important weapon in Ioannidis's myth-busting arsenal, and I think the proliferation of statistics within experiments has had a positive effect on science.

On the other hand, I think the proliferation of epidemiology, which uses even more advanced statistics, has in some ways had a negative impact because scientists and media people are too tempted to inflate the importance of these types of studies by abandoning everything taught in Stats 101 and Epi 101 like “correlation does not imply causation.”

Nevertheless, despite the rampant peddling of hypothesis-as-fact, statisticians are increasingly developing methods to detect publication and reporting bias.  These will be critical to cutting through all the junk to find the truth.

Back to EGCG for a minute.

It's important to realize that every experiment has some sampling error.  Statistical tests designed to determine whether a difference is likely to be real (“statistically significant”) or just a random fluke are based on the concept of a sampling distribution.  The sampling distribution is based on the idea that if you repeated an experiment a hundred times, you'd get a hundred different results, but the results should all hover around the “true” value.

For example, let's say that the “true” effect of EGCG on uptake of glucose into skeletal muscle is zero, zip, zilch, nada.  It just doesn't do anything.  In that case, the “true” difference in glucose uptake between skeletal muscle cells that have been treated with EGCG and those that haven't should be zero.  Thus, we can construct a theoretical sampling distribution that has a mean, or average, of zero:

The sampling distribution will be a bell-shaped curve that has as its center the “true” value, and we are offering the hypothesis that this value is zero because we are testing the hypothesis that EGCG has no effect.

The bell-shaped curve represents all the possible results of experiments.  As the blue line rises, experiments are more likely to come up with that result.  The curve is shaped like a bell, that is, raised in the middle, because results closer to the “true” effect — which we are hypothesizing is no effect, or zero — will be more common than results further away from the true effect.

Just how wide the curve is will depend on the statistical precision of the experiments. The greater the sample size and the lower the variability, the narrower the curve will be, meaning each experiment is likely to give a result closer to the “true” result.  It will generally stretch three standard deviations in either direction.  The standard deviations are marked off in units along the horizontal axis.  The standard deviation of this distribution is usually called the standard error.

For the sake of simplicity, let's say that if we repeat the experiment 100 times, 95 times we'll get a result within two standard errors.

Now let's say EGCG really affects uptake of glucose into muscle.  If it falls outside of that range, in the tails of the distribution, we say the result is statistically significant because the chance of getting that result if EGCG truly has no effect is less than 5%.

But, what if negative results never get published?  Say there are five papers published on the topic.  What if 100 people tried the experiment and 95 of them got no result so never published their findings?  EGCG could have no effect, but what you might see is three papers showing a positive effect and two showing a negative effect, and 95 experiments lost down the memory hole, never making it further than some lab technician's notebook paper.

Thankfully, there are some incentives against this phenomenon.  Among them, researchers don't want to waste time and money failing to generate publications.  There are a great many papers suggesting that green tea can help prevent diabetes, obesity, and fatty liver disease.  As a result, many labs will dedicate themselves to studying this effect and trying to explain it.  This will allow them to mention negative findings to “rule out” mechanisms as part of larger papers with positive findings that render the paper “interesting” to a journal editor.

Nevertheless, publication bias is likely to be a serious problem, especially in clinical trials where there is a strong incentive to show that a drug or other treatment has an effect.

Statisticians have developed a cool tool called the funnel plot in order to detect publication bias.  Here's a recent example of a funnel plot showing clinical trials of long-chain omega-3 fatty acids EPA and DHA in the treatment of depression:

In the above diagram, all the white circles represent trials that were actually published.  As you move up along the vertical axis, the statistical precision increases.  The plot should be funnel shaped and appear symmetrical, so that studies with more statistical precision hover more tightly around the average and studies with less statistical precision gradually spread out toward a base at the bottom.

However, as you can see above, the distribution of white circles is not symmetrical.  This indicates publication bias.  In order to construct the funnel plot, the authors trim off the asymmetrical portion and then draw a line down the middle of the symmetrical portion.  This is the estimated “true” mean of the trials.  Then, the authors estimate how many trials weren't published by throwing onto the diagram enough black circles to make the plot a symmetrical funnel shape.  These black circles represent hypothetical “unpublished trials.”

Here, the authors estimated that there were nine trials that found EPA or DHA actually increased depression scores but were never published.

Obviously, there is something inherently impossible to validate about this method.  We can never know what is being kept in secret, so we can never actually prove this method works.

Nevertheless, there are many methods we use in the looser sciences that we cannot validate.  For example, food frequency questionnaires (FFQs) can be kind-of-sort-of validated with weighted dietary records, but people may eat differently when they are weighing their food, and weighted dietary records can never be done over the full course of time that an FFQ is meant to apply to, often a year.  It is even more difficult, in fact completely impossible, to validate radiometric dating over the time course we use it for.  But we use reasonable assumptions to make estimations with these tools because that is better than throwing our hands up in the air and giving up.

So while funnel plots can never prove publication bias, I hope that they will bring the probability of publication bias to light, and thereby encourage the scientific community to be more transparent.

One of the moves towards transparency currently happening is to create a database of clinical trials where they are registered before they are conducted.  This is an incredibly important development that will help prevent publication bias.

Another important bias is reporting bias.  If the researcher measures 100 things and reports five of them, then how much credibility can we give to the “statistical significance” of those five things?  If you measure 100 things, you are bound to get five or so that are “statistically significant” but are really random flukes.

Statisticians are currently working on methods to detect this type of bias in reporting.  I have no idea how they'll work, but I can't wait to find out.