The accompanying editorial was written by Barry Popkin, an economist who used to work for the Rockefeller Foundation and is currently a fellow of the Carolina Population Center, an organization started by the Ford and Rockefeller Foundations in the 1960s as part of their effort to institute worldwide population management programs. Popkin recommended reducing saturated fat to less than seven or ten percent of calories, requiring “higher-income countries to significantly cut their animal source food intake, shift to leaner meats, and shift to reduced-fat dairy products.” Strangely, he left out any discussion of the relationship between low-fat dairy products and infertility; this relationship would certainly make switching to low-fat dairy products a good form of population control. He concluded that practicing clinicians should avoid preaching vegetarianism but rather advise a general reduction in the intake of animal foods, to prevent chronic disease and global crises of food, water, and climate.
News reports claimed that the study showed eating the equivalent of a quarter-pound hamburger per day “gave” men and women 20-50 percent increases in the risk of chronic diseases such as cancer and heart disease. In reality, the study showed no such thing. It was not designed to determine cause and effect, and its ability to determine true meat intake was almost non-existant. News reports and editorials alike failed to discuss its embarassing finding that meat intake was associated with the risk of dying from accidental injury, probably because the apparent lack of a plausible mechanism by which eating meat could cause someone to get into a car accident emphasizes the most basic principle of science that they want us all to forget: that correlation does not prove causation.
There are thus two important points we need to understand about this study to realize just how little it does to increase our knowledge:
- The study found a correlation between increased mortality and a population’s propensity to report eating meat, not a correlation between mortality and true meat intake. As we will see below, these may be two completely different things.
- Correlation does not show causation. There is absolutely no scientific basis to conclude from this study that eating meat increases mortality.
These types of studies are useful to generate new hypotheses, but this study failed to do even that. As we will see below, it therefore provides us with no useful information of any kind.
Reported Meat Intake — Is It Related to True Meat Intake?
Beginning in 1995, the authors of this study gave a food frequency questionnaire (FFQ) to over a half million people. You can view the questionnaire here. It contained 124 questions, each question about a particular type of food or group of food. It asked the participant how often she or he consumed the food over the course of the last year, giving them ten options. Then it asked how large of a serving size they consumed, giving them usually three or four options. Sometimes they were given additional instructions, like including sandwiches in some cases or excluding sandwiches in other cases.
Obviously any individual trying to quantify his or her average intake of 124 foods over an entire year is going to have to engage in a lot of guess work. Even 24-hour recalls of what a person ate the day before are subject to a great deal of error. For this reason, researchers will commonly “validate” an FFQ or a 24-hour recall to test whether these accurately measure the intake of the foods of interest. In order to do this, they have the participants make a weighted dietary record where they meticulously weigh everything they eat with a dietetic scale and record it as they prepare each meal. Then the researchers compare the FFQ or 24-hour recall to the weighted dietary record, assuming that the weighted dietary record is the best indicator of true dietary intake.
The researchers who published the red meat study did this differently. They “validated” their FFQ with two 24-hour recalls!
This is rather strange. Consider how the researchers who conducted the Nurses’ Health Study described the need for validation:
The role of diet in disease processes is of great scientific interest. One limitation in assessing associations between diet and disease is the often-large measurement error in reported dietary intake, which arises from two major sources. First, there is random within-person variation in reported dietary intake based on commonly used instruments such as a food frequency questionnaire (FFQ) or a 24-hour recall. Second, even if a surrogate instrument (e.g., FFQ) were perfectly reproducible, it might not be a valid measure of true dietary intake as might be captured in a weighed diet record, in which subjects record what they eat on a real-time basis.
To address the validity issue, it is becoming common to conduct a validation study. A small subset of persons, ideally from the same population as the main study, are administered both the surrogate instrument (e.g., FFQ) and a “gold standard” instrument (e.g., diet record) and the relation between them is ascertained.
If the FFQ and the 24-hour recall are both “surrogate instruments” in need of validation with a directly measured dietary record, how can a 24-hour recall instead of a dietary record be used to validate an FFQ? Obviously, it can’t.
Moreover, the authors’ “validation study” found that the “true intake” of protein, carbohydrate, fat, cholesterol, fiber, vitamins, minerals, fruits, and vegetables estimated by the 24-hour recall could explain between 5 percent and 45 percent of the variation in the participants’ answers on the FFQ, but they never “validated” the FFQ’s ability to predict the “true intake” of meat!
It is therefore bizarre to say the FFQ was “validated” at all.
Fortunately, the researchers who conducted the Nurses’ Health Study (Harvard) conducted it far more rigorously than the researchers who conducted this study (the Federal government), so their research can give us an idea of just how foolish it is to believe that answers to questions about meat on an FFQ represent true intake of meat.
Researchers conducting the Nurses’ Health Study have administered and continue to administer FFQs repeatedly to the study’s participants to track food intake over time. They validated their FFQ with four seven-day weighted dietary records, each given three months apart. Their validation therefore takes into account day-to-day variation within the week as well as seasonal variation during the year.
They found that the FFQ predicted true intake of some foods very well and true intake of other foods very poorly. True intake of coffee could explain 55 percent of the variation in answers on the FFQ, while true intake of beer could explain almost 70 percent. True intake of skim milk and butter both explained about 45 percent, while eggs followed closely behind at 41 percent.
But the ability of the FFQ to predict true intake of meats was horrible. It was only 19 percent for bacon, 14 percent for skinless chicken, 12 percent for fish and meat, 11 percent for processed meats, 5 percent for chicken with skin, 4 percent for hot dogs, and 1.4 percent for hamburgers.
If your jaw just dropped, let me assure you that you read that right and it is not a typo. The true intake of hamburgers explained only 1.4 percent of the variation in people’s claims on the FFQ about how often they ate hamburgers!
What explained the other 98.6 percent of their answers on the FFQ? One possibility for which there is substantial evidence is that some foods are, in our culture, socially and emotionally charged, and participants are more likely to lie about their intake of those foods, or more likely to deceive themselves about how much of those foods they are really consuming. Consider what the researchers who validated the Nurses’ Health Study FFQ had to say:
Focusing on the second questionnaire, we found that butter, whole milk, eggs, processed meat, and cold breakfast cereal were underestimated by 10 to 30% on the questionnaire. In contrast, a number of fruits and vegetables, yoghurt and fish were overestimated by at least 50%. These findings for specific foods suggest that participants over-reported consumption of foods often considered desirable or healthy, such as fruit and vegetables, and underestimated foods considered less desirable. . . . This general tendency to over-report socially desirable foods, whether conscious or unconscious, will probably be difficult to eliminate by an alteration of questionnaire design.
The implications are even worse than suggested in this quote. The authors looked at the results from two FFQs, one administered three to six months before the first dietary record was taken, and a second one administered a year later, on the immediate tail of the third or fourth dietary record. In other words, the results from the second FFQ should be more accurate because the participants had just spent a week meticulously measuring and recording everything they ate. So the tendency to exaggerate intakes of socially desirable foods and under-report intakes of socially undesirable foods was probably even greater on the first FFQ, and thus even greater on an FFQ in the current NIH-red meat study, in which no dietary record was ever administered.
If true intake of hamburgers represents 1.4 percent of the variation in intakes as estimated by the FFQ, and other factors such as a person’s willingness to deceive themselves about how much hamburger they are eating or, worse, lie about how much hamburger they are eating explains at least a substantial portion of the other 98.6 percent, would you believe any study finding correlations between hamburger consumption and some health endpoint was actually testing the effect of eating hamburgers? I wouldn’t.
We must therefore be careful to say that the NIH study found an association between increased mortality and someone’s willingness to report consuming red meat to researchers from the Federal government. We must carefully avoid saying that the study found an association between mortality and actually eating red meat.
Remember the Scientific Method — Correlation Does Not Prove Causation
Every student learns this most basic fact of science in any introductory science course: correlation does not prove causation. If A increases as B increases, this may be because A causes B, but it may also be because B causes A or because a third factor C causes them both.
The NIH study found that people who reported eating more red meat were more likely to be married, to be white, to smoke, to eat more food in general, to weigh more, to be less educated, to take vitamin supplements less often, to eat fewer fruits and vegetables, and to exercise much less often. In fact the proportion of people that engaged in vigorous physical activity among those who reported high intakes of red meat was only 16 percent, hardly more than half of the proportion of people that engaged in vigorous physical activity among those who reported low intakes of red meat, which was 30 percent.
Researchers call these confounding variables and use statistical methods to try to control for them. But there are always other confounders that the researchers do not know about. For example, this paper did not report statin use among participants. Since these drugs are extremely common among elderly people and are believed to affect lifespan, could they not present us with another confounding variable? Are people who report eating more red meat more or less likely to be placed on statins?
The potential number of confounders is limited only by our imaginations. This is why we never, ever use epidemiological studies that uncover correlations to demonstrate causation.
To illustrate the pitfalls of confusing correlation with causation, let us consider this: the study found a 26 percent increase in the risk of “mortality from injuries and sudden deaths” among those who reported eating the most red meat. It is entirely possible that eating red meat could increase the risk of injury, but is it probable? Not very. So the news reports and editorials have largely ignored this finding in favor of the increase in heart disease and cancer risk it found, because it seems intuitively more reasonable to most people that eating meat could cause these diseases.
The strongest correlation, however, was between eating red meat and the risk of “all other deaths,” a category that included mortality from tuberculosis, HIV, other infectious and parasitic diseases, septicemia, diabetes, Alzheimer disease, stomach and duodenal ulcers, pneumonia and influencza, chronic obstructive pulmonary disease and related conditions, chronic liver disease and cirrhosis, nephritis, nephrotic syndrome and nephrosis, congenital anomalies, certain conditions originating from the perinatal period, ill-defined conditions, and unknown causes of death.
In order to begin laying down a solid hypothesis that reporting eating red meat (which is falsely assumed to represent actually eating red meat) actually causes death from all of these diseases, the authors need to make an argument that causation is biologically plausible. And that is just to present the hypothesis. In order to begin confirming the hypothesis, studies in cells, animals, and eventually humans are needed to show that modifying red meat intake controls the risk of these diseases.
The authors have another large obstacle before them: reporting a high intake of white meat was associated with a lower risk from almost all of these deaths! But the chemical constitutents of red meat that the authors of this paper blame for causing disease — carcinogens formed during high-temperature cooking, iron, and saturated fat — are also found in white meat.
Epidemiological studies such as this one are observational in nature. Observation is the first step of the scientific method. One then develops a hypothesis to explain those observations, and then tests the hypothesis through experimentation. Epidemiological studies, therefore, are useful for generating hypotheses, but not for testing them. They could also be used to quantify the importance of a risk factor once it is already established to cause the health-related endpoint with which it is associated through experimentation.
Red meat has not already been shown to cause early death, so this study cannot be used to quantify its importance as such a cause. The hypothesis that red meat causes all sorts of disease already exists, so this study cannot be used to generate a new hypothesis.
If this study had analyzed red meats of different quality separately, it may have shed light on a new hypothesis. It could have analyzed grass-fed versus grain-fed meat. It could have analyzed meat from animals raised on pasture grown in soils of different quality. It could have analyzed how well the meat was cooked. But it did none of these things.
The study therefore provides us with no more useful information than we already had. In fact, by using an unvalidated food frequency questionnaire that most likely had almost no ability to actually estimate true intake of red meat, the study just adds more confusion to the multitude of conflicting claims about the relationship between diet and health. This is where our tax dollars are going to, at least the portion of them that is not going towards bailing out the big banks. Meanwhile, I will continue to eat red meat.