Some Problems with Anecdotal Evidence

One of the major pitfalls of nutrition is that anecdotes are often given equal footing with scientific evidence. Stories of how you can use diet to rid yourself of acne, gain more energy, or stop that persistent runny nose proliferate in the uncritical halls of the web where they can almost always find an eager audience.

Unfortunately, these stories are often no more than that—stories. The plural of “anecdote” is not “data”, and when it comes to scientific validity, no amount of anecdotes can compete with a single well-done study.

A while back, I worked for someone who loved anecdotes but also claimed to also truly care about scientific evidence. One of my jobs was to take her anecdote-based ideas and see what the research said about them. Whenever the research backed her anecdotes, she would enthusiastically use it as evidence that her idea was correct—but when the research was unsupportive, anecdote trumped science.

One day she said to me, “Wow, you really only listen to what the scienctific evidence says, huh?” She was incredulous that I didn’t put stock in success stories because, to her, the proof was right there and scientific evidence was really only useful to back what she felt in her gut. When the evidence didn’t confirm her anecdotally-driven hypotheses, she thought it was due to a problem with the science, not her story.

This exchange happened close to the day I quit; I think we both had realized there were irreconcilable differences in our approach to evidence. But as frustrating as I found her approach, it also made me realize that a fundamental distrust in anecdotal evidence is a learned skill, and that if I really wanted people to understand why nutritional anecdotes cannot take the place of nutritional science in healthcare then I would need to educate them—so here we go!

Anecdotal Evidence Favors Resolution

Humans love stories—I love stories! Stories unite us (and divide us), they entertain us, and they guide us. Stories can be powerful.

When something bad happens to us, it’s natural to try to explain it with a story—and when we find a story that fits our experience, our first instinct is to embrace it, not question it. This is particularly true when the story comes from a trusted source like a friend, a relative, or even ourselves (because yes, we all tell ourselves stories). But the very thing that makes a story good is also what makes it bad as evidence: stories favor resolution.

Nobody wants to read a story that goes nowhere, but sometimes the honest truth in nutrition is that we don’t know what the problem is—or we do know the problem, but also know it cannot be resolved through nutrition. This doesn’t make for a compelling narrative, and such a narrative cannot compete with online success stories. It’s like reading a story where the young boy who knows he’s destined for greatness never does anything particularly interesting with his life—who wants to read that?

When we read stories—even the simplest stories about nutrition and health—we expect catharsis. We want the story to reveal our inner dragons and then show us they can be slain. Good stories resonate with you not only because you relate to the problems the protagonist faces, but because the protagonist shows you those problems can be solved. If a story doesn’t live up to this standard, then it is shunned and forgotten—or in modern terms, it goes unshared and untweeted.

Thus, the stories that inevitably float to the surface are the positive success stories, not the tales of disappointment, frustration, and failure. Only good stories get retold, and a story that doesn’t resolve isn’t a good story.

If you want an unbiased account of what works and what doesn’t in nutrition, you have to ignore the story and seek out impartial science. The scientific method has no comfortable “obstacle, conflict, resolution” story arc, and it doesn’t care about catharsis. Yes, every scientific study attempts to answer a question, but in pursuit of that answer you often find that you were wrong.

In fact, when formulating a scientific hypothesis, the base assumption (called the “null hypothesis”) is that you’re wrong and that there is no real relationship between the variables you’re studying. When a study fails to demonstrate a relationship, you “accept” the null hypothesis (you accept that your hypothesis is incorrect) and you move on to a new one. Compare this to a good story, where the null hypothesis—the base assumption at the very beginning—is that everything will work out.

This is a major problem with anecdotes: they will always favor a positive solution—or in the case of “cautionary tales”, a negative one. Good science doesn’t favor any solution, and it proceeds with the basic understanding that you’re going to be wrong much of the time. Yet this willingness to be proven wrong is responsible for 100% of modern progress in health, medicine and nutrition. As the adage goes, you don’t learn from success, you learn from failure.

Anecdotal Evidence Is Contradictory

A second problem with anecdotal evidence is that there are always at least two sides to the story. Usually when we say this, we mean that two people in an altercation each have their own version—but in this case I mean that you can always find an anecdote contrary to your own pet belief.

If you don’t believe me, all you have to do is Google the opposite of something in which you wholeheartedly believe. To be sure, there is a usually a dominant narrative and a less-dominant one—but if you’re only looking for evidence that you’re right (by Googling exactly what you believe and not what you disbelieve), then that’s the only evidence you’ll ever find.

There’s a reason why courts use physical evidence (inasmuch as they can) to resolve disputes—the story that each party tells is guaranteed to be contradicted by the other party, which is the whole reason they’re in court in the first place. Physical evidence helps establish facts, and facts supersede anecdotes, which allows the judge to make a (hopefully) fair ruling.

Unlike the legal system, the scientific method has the ability to completely rule out anecdotes in pursuit of the truth, and we can therefore have even better assurance that what we’ve concluded is likely to be accurate (especially when later repeated). Anecdotes may spur a scientific inquiry, but they give way to evidence once it’s presented.

Rejecting scientific evidence in favor of an anecdotal account is akin to rejecting physical evidence during a trial in favor of the defendant’s story. Yes, there’s a chance your conclusion may later be proven wrong, but protestations of innocence hold far less weight than DNA samples and matching boot prints.

Anecdotal Evidence Is Incomplete

Anecdotes focus on a single reason—or, at best, a group of related reasons—for why something works. As a result, they cannot help but ignore the myriad other reasons for why something may have happened. In science, this is called confounding and it can skew results terribly.

A great example of this can be witnessed whenever a friend starts a new diet and claims to feel better, to which they of course credit the specifics of the diet, whether that means avoiding gluten, or carbs, or fat, or whatever. Ignoring the placebo effect for the moment, the problem is that anytime someone starts a new diet they make numerous changes to their diet—and most of the time they’re just plain healthy changes.

For example, what do the Atkin’s diet, the Mediterranean diet, and the Ornish diet have in common? None of them allow junk food, and all of them encourage greater veggie consumption. Eating fewer chips and more vegetables is going to improve your health regardless of whatever else the diet specifies—but the dieter’s rationale for why they feel better isn’t that their diet has generally improved, but rather that they’ve adhered strictly to the “rules” of the diet.

Anecdotes are almost always “accidentally incomplete” for this reason—because the storyteller cannot control or account for all of the variables. But sometimes anecdotes are purposefully incomplete, as was recently demonstrated by the story of Belle Gibson. If you didn’t follow the news earlier this year,  Belle Gibson was an Australian “health guru” who claimed to have cured her brain cancer with a healthy diet and alternative medicine. She even released an app (“Whole Pantry”) to help others do the same, and made at least $300,000 from it.

Her story was convincing, especially to those with terminal cancer who may have felt no hope—but what her followers didn’t know was that she had purposefully withheld a fundamental part of her story, which is that she never had cancer. Yes, most of us would call this a blatant lie, but I worded it as “withheld” because it belies an oft-ignored truth: when it comes to anecdotes, you never know what you don’t know, and you can never know if you know the truth. That’s a tongueful, but simply put it means you’re not privy to the storyteller’s notes.

Not everyone is an outright liar like Belle Gibson, but the reality is that most people will embellish facts and blur truths in pursuit of a compelling narrative, and they may not even realize when they’re doing it. Our minds are less concerned with the facts of a story than whether it fits our worldview, and will subconsciously alter our recollection to retroactively fit it, which makes anecdotes unreliable even when they come from otherwise reliable sources.

When you hear an anecdote, it’s important to keep all of this in mind. What hasn’t the storyteller accounted for? What have they forgotten or exaggerated? What are they purposely not sharing? The answers could—and very likely would—change how you feel.

Don’t Trust Anecdotes; Trust Science

Anecdotes are not scientific evidence, they’re carefully crafted stories designed to hook you. There are reasons to distrust anecdotes beyond the problems I’ve brought up in this article, but the three covered above are major.

In summary, anecdotes are unreliable because…

1. They favor positive, interesting resolutions (selection bias).
2. The anecdotes you hear are far more likely to confirm what you already believe than contradict it (confirmation bias).
3. You never have the whole story (confounding, embellishing, and lying).

When science is done right, these factors are either eliminated or controlled for to as great an extent as possible. When a scientific study suggests that the majority of people with an intolerance to gluten actually do not, it’s worth a lot more than a mountain of gluten-related anecdotes. This is especially true since such studies also direct research to more appropriate hypotheses (like FODMAPs, in the case of gluten)—that is, hypotheses that might actually help people rather than hold on tight to a disproven idea.

So stick with science and be skeptical whenever you hear a nutrition-related anecdote. They may make for a great success story, but they tell you almost nothing about what really happened.