Anecdotal Evidence Does Not Make Good Policy

Recently, a series of anecdotes appeared on the website People’s Pharmacy, posted by readers who suffer from depression. They detail how switching from name brand antidepressants to generics—or in some cases from one generic to another—caused a return in symptoms. Here’s one excerpt:

“I've been on the big yellow 300 mg Buproprion XL for about 6 months and my last refill was filled with Actavis (smaller white ones). While the old ones…made me feel great, I've been doing poorly on Actavis…I certainly feel more unmotivated, lethargic, and sadder.”

People’s Pharmacy isn’t connected to the Food and Drug Administration, a medical center or a mental health or other professional group. Joe Graedon, the site's founder is a consumer advocate and writer, not a health professional. But nevertheless, the New York Times prominently quotes him and includes similar anecdotes in a recent article that questions whether generic drugs are as effective as their name-brand counterparts.

The unfortunate title of that piece, “Not All Drugs Are the Same After All,” was “pretty irresponsible,” says Aaron Kesselheim, an instructor in medicine at Harvard Medical School who has authored studies comparing generic and name-brand drugs. For one, it doesn't reflect the body of evidence supporting the use of generics. “For the vast, vast, majority of pills and the vast, vast majority of patients there is no evidence of problems substituting generics for brand name drugs.”

There is ample evidence that when patients are prescribed generic versions of proprietary drugs they are far more likely to actually fill their prescriptions, take their medications and far less likely to split or skip doses. The reason, of course, is affordability: Generic drugs cost 80-85% less than their name-brand counterparts and they make up 7 in 10 prescriptions currently filled in this country.

The “controversy” over generic drugs is really less than meets the eye. There have been scattered concerns—some based on patient or doctor reports, others from proprietary drug makers—about generics not being biologically close enough (bioequivalent) to name brand meds. But the issue is mired in complexity and misinformation. For example, many doctors accept significant payment from drug companies to promote their name brand drugs—especially cardiologists and psychiatrists. It can be hard to remain unbiased when a pharmaceutical company is paying the tab on research.

In the case of depression, the Times neglects to report that pharmacological treatment is often based on trial-and-error and is plagued by uncertainty. Just for starters, the placebo effect of antidepressants drugs is quite large. In fact, in a study published last week in the Journal of the American Medical Association researchers report that “[t]he magnitude of benefit of antidepressant medication compared with placebo…may be minimal or nonexistent, on average, in patients with mild or moderate symptoms.”

Not surprisingly, this placebo effect, while strong and certainly measurable, can be related to the size, shape and color of the pill—all of which differ between manufacturers of the same active ingredient. Drug companies know this and, according to Generics Are Powerful Medicine, a consumer education program; companies are increasingly trademarking not just the name but also the appearance of proprietary drugs—Nexium’s purple-with-yellow stripes and Viagra’s blue diamond shape are two examples. Trademarks—unlike patents which last a maximum of 20 years—never expire and companies can continue to capitalize on the connection they’ve actively promoted between a drug’s appearance and its effectiveness long after losing patent protection.

All of this points to the pitfalls of using anecdotes to justify changes in drug policy. The fact is, in the overwhelming majority of cases, studies find that generic drugs work just like their name-brand counterparts. The active ingredient must be identical and in the same concentration. Granted, inactive ingredients like binders and colorings can vary. But studies like this one in The Annals of Pharmacotherapy that reviewed 12 years of data from the FDA, find name-brand and generic drugs to be bioequivalent. The Annals study found that the average difference in absorption into the body between generic and name brand drugs was just 3.5%; similar to the difference found between batches of the same name brand drug.

Hesselheim, along with colleagues at Brigham and Women’s Hospital in Boston, published this review in JAMA  that looked at 38 published clinical trials comparing cardiovascular generic drugs to their brand-name counterparts. Their findings: There was no evidence that brand-name heart drugs worked any better than generic heart drugs.

So why is controversy growing over generic drugs right now? For years insurers—both private and public—have used formularies that encourage consumers to fill prescriptions with generic drugs. This has saved billions of dollars in health care costs and been one of the few bright spots in a growing spending crisis. The trend towards substituting generics for far more expensive name-brand drugs is also a clear example of how comparative-effectiveness studies can lead to cost savings.

A lot of lip service has been paid to the value of this approach and the government has already provided $1.1 billion for comparative-effectiveness research through the American Recovery and Reinvestment Act,  and both the House and Senate reform bills would fund centers to promote this kind of work.

Unfortunately, the public—encouraged in part by drug companies, doctor’s groups and other ingrained interests—is still somewhat wary of comparative-effectiveness studies (and the related field of evidence-based medicine that uses such studies to develop treatment guidelines). To some, this approach is a thinly-veiled method for instituting health care rationing. Conservatives capitalize on this fear, raising the specter of government panels using such studies to mandate certain types of care; of the government “getting between you and your doctor.”

Increasingly, the powerful anecdote has emerged as the most effective weapon in fending off evidence-based medicine. Patient anecdotes provide a basic source of information for doctors—they are integral to the emerging discipline of narrative medicine that stresses the importance of stories in the illness experience. They are the antidote to “checklist medicine,” reminding practitioners that patients are complex individuals and that one person’s experience of breast cancer, for example, can vary quite significantly from another's.

But anecdotal evidence should not be the basis for policy changes. This became all too clear in recent months when the U.S. Preventive Services Task Force came out with their new recommendations on routine mammography screening. When the group failed to recommend routine screening for women under 50, powerful anecdotes from women who were convinced that early screening saved their lives or the life of someone close to them rose to the forefront.

The Task Force recommendations were quickly refuted by the cancer establishment, as well as many physicians and survivor groups. Despite the very real body of evidence indicating that mammography is overused and can lead to significant false-positives, over-diagnosis and over-treatment of breast cancer, the anecdotes won out. The issue of breast cancer is emotionally charged and going against a long-standing campaign that continues to equate screening with cancer prevention is no easy job.

We’ve seen the power of anecdotes in other controversial medical areas. Take the recent consensus report in the journal Pediatrics,  authored by a panel of experts, which found that special diets do not benefit autistic children. Specifically the report states:

“Available research data do not support the use of a casein-free diet, a gluten-free diet, or combined gluten-free, casein-free (GFCF) diet as a primary treatment for individuals with ASDs (autism spectrum disorders).”

When reporting on this study, ABC News decided to bring on an “expert” who would provide balance for the story by casting doubt on the study design and its findings. As Forbes’ David Whelan at The Science Business reports, ABC News chose Jenny McCarthy—actress, Playboy model and mother of an autistic son—as that “expert.” McCarthy’s comments feed directly into this cult of the anecdote that drives many medical news stories and, increasingly, health policy:

"Until doctors start listening to our anecdotal evidence [about the success of special diets], which is it's working, it's going to take so many more years for these kids to get better," she said.

McCarthy, of course, has used this same anti-scientific logic in her frequent attacks on childhood vaccines; advancing anecdotes about how her son and other young children were completely normal until vaccinated as proof that shots cause autism. This line of thinking continues despite the many studies, including a new report that will appear in the May 2010 Pediatric Infectious Disease Journal, supporting the assertion that MMR and other vaccines do not cause autism in children.

The cult of the anecdote extends beyond non-medical types like Jenny McCarty and journalists. Norbert Gleicher, president and medical director of the Center for Human Reproduction in New York City, uses an anecdote about his mother to support his convoluted attack on evidence-based medicine in an editorial in the Wall Street Journal:

“[E]xpert panels would only slow medical progress and delay rejection of false prophecies and dogmas. These panels would interrupt a well working free market of ideas in health care, where effective therapies can rise to the surface and win out.

 “Of course evidence-based data can be useful. But I have seen firsthand how it can be dangerous.

“Several years ago I grew concerned about my postmenopausal mother's risk of osteoporosis. I tried to convince her to initiate hormone replacement therapy. She didn't listen to me. Instead, she spoke with her gynecologist, who—contrary to best medical evidence at the time—recommended against such treatment. I would eventually be thankful my mother listened to the gynecologist who had known her for decades instead of me and the published medical reviews I was relying on. Some years later my mother was diagnosed with early breast cancer. Had she been on estrogen replacement, it is likely that her tumor would have progressed more rapidly. The gynecologist likely saved my mother's life.”

Gleicher’s story about his mother, ironically, could be used just as powerfully to support the need for more comparative-effectiveness studies. The push to give so many post-menopausal women hormone replacement therapy was primarily a function of this “free market of ideas,” i.e. intensive marketing by drug companies—not evidence-based medicine. Once evidence from well-designed, long-range studies pointed to some of the dangers of HRT, an expert panel made new, more conservative recommendations about its usage.

Where do we go from here? The goal of evidence-based medicine is not to ignore anecdotes. Sometimes there are enough consistent reports that anecdotes can reach a critical mass. When doctors begin reporting noticeable changes in symptoms, concrete measures of disease or pathology, then anecdotes begin to build hypotheses. When a well-designed study generates data that can be analyzed, and this effect is found to be significant, then we have scientific evidence.

In the case of generics, doctors have reported enough incidents of breakthrough seizures in some epileptics who switch from name-brand to generic versions that Congress has asked the FDA to study the issue and determine whether changes need to be made in bioequivalency standards for this specific class of drugs. These anti-seizure drugs are called “narrow therapeutic index drugs” which, according to Kesselheim means that there is less “wiggle room” between effective and toxic doses and seizure control is “tenuous.”

The FDA’s findings may ultimately lead to tighter bioequivalency standards being set for this one class of drugs. And that is how evidence-based medicine is supposed to work. But observations do not make health policy—neither should “gut feelings,” political causes or industry profits. There’s too much at stake.

Giving credence to claims by vaccine opponents that childhood immunizations cause autism leads to loosening of vaccine mandates and the emergence of once-vanquished diseases. Demanding that insurers and government programs pay for name-brand drugs when there is no hard evidence of difference in effectiveness drives up health care costs for everyone. And by ignoring the recommendation that many women under 50 may not need yearly mammograms, we ensure that over-diagnosis and over-treatment for cancer continue to plague the field. The anecdote has its place in medicine--just not in forming policy.

Comments

[Peter P]

What a well-written and intelligent piece.

[Rosiecee]

A scientific study out of Columbia University released today showed that half of the people who take antidepressants are having their serotonin neurotransmitters lowered instead of raised as is suppose to happen.

Perhaps that is why thess 3,500+ anacedotal stories on www.SSRIstoies.com [from the media] are appearing.

The Physicians Desk Reference states that SSRI antidepressants and all antidepressants can cause mania, psychosis, abnormal thinking, paranoia, hostility, etc. These side effects can also appear during withdrawal. Also, these adverse reactions are not listed as Rare but are listed as either Frequent or Infrequent.

Go to www.SSRIstories.com where there are over 3,500 cases, with the full media article available, involving bizarre murders, suicides, school shootings/incidents [51 of these] and murder-suicides - all of which involve SSRI antidepressants like Prozac, Zoloft, Paxil, etc, . The media article usually tells which SSRI antidepressant the perpetrator was taking or had been using.

[tony bateson]

Before the new report clearing vaccines of links to autism etc. appears in Pediatric Infectious Disease Journal in May 2010 the authors would do well to contemplate the UK facts. If they could show me that the UK population of some three millions who have not had childhood shots contains any autistic people whatsoever I might be willing to read their otherwise unbelievable conclusions.

More than ten years of sometime approaching industrial strength searching has so far failed to deliver up any evidence that unvaccinated autistic people exist in the UK. If they are not here it is laughable that vaccines are not linked to autism.

Tony Bateson, Oxford, UK.

[John Libonati]

You incompletely report the findings of the Pedatrics article on Gastrointestinal Disorders in Autism. I read it and can tell you a thorough read of the medical panel statement, instead of relying on ABC's inaccurate report, will lead you to a far different conclusion. Only 15 children were included in the doubleblind study to determine effectiveness of the gluten-fre, casein-free diet on autism. You should be well-aware that is a pilot study and too small a sample upon which to base a final conclusion. And what about the part where the researchers stated, “Nevertheless, after being informed of the results, 9 parents wanted to continue the diet and reported positive subjective clinical changes while their child was on the GFCF diet." Read my article below to find the full text, what the panel really said, and how the media is fueling the gluten-free, casein-free autism treatment controversy. http://blog.glutenfreeworks.com/2010/01/06/is-the-media-creating-the-gluten-free-casein-free-autism-controversy/