Medicine’s Bad Data Problem

Steven Cherry Hi, this is Steven Cherry for Radio Spectrum.

The saddest fact about the coronavirus pandemic is certainly the deaths it has already caused and the many more deaths to come before the world gets the virus under at least as much control as, say, chicken pox or an ordinary flu.

The second-saddest fact about the pandemic is the economic and educational havoc it has wrought.

Perhaps the third-saddest fact is the unfortunate lack of agreement about the best strategies for living with the virus, which, at least in the U.S., is responsible for many of those deaths, and, arguably much of the havoc as well. It has roiled families as well as the presidential election, by politicizing the wearing of masks, the limits on gatherings, the openings and closings of restaurants and schools.

But yet another sad fact is that, as was said thousands of years ago, “there is nothing new under the sun,” and this too is nothing new; there is a shocking and unfortunate lack of widespread agreement about the best answers when it comes to many medical questions, even among doctors, because there is a shocking and unfortunate lack of evidence—and even respect for evidence—in the medical arena. That’s the contention of the authors of a rather prescient 2017 book, Unhealthy Politics: The Battle Over Evidence-Based Medicine,subtitled, how partisanship, polarization, and medical authority stand in the way of evidence-based medicine.

And so I’ve asked one of those authors to lay out the case that medicine isn’t nearly as evidence-based as we think it is and as it should be, and tell us what role that has played in the severity of the pandemic—and what we might do about it, both in the near pandemic future, and to improve American medicine overall.

Eric Patashnik is a professor of public policy and political science at Brown University and the director of its Master of Public Affairs Program. He joins us via Zoom.

Steven Cherry Eric, welcome to the podcast.

Eric Patashnik Thank you so much for having me. It’s a pleasure to be here.

Steven Cherry Eric, the book starts with a rather surprising fact about how little we know about an operation that’s performed millions of times in the U.S., which you call a sham procedure. What is the sham procedure and how is it possible that a procedure that’s known to be a sham can be performed millions of times?

Eric Patashnik Yeah, so the original motivation for my book, which is coauthored with Alan Gerber at Yale and Conor Dowling at the University of Mississippi, was a remarkable study published in the New England Journal of Medicine in 2002. And what the study found was that a very common surgical procedure, arthroscopy for osteoarthritis of the knee—which is an operation that many older folks who have arthritis get after they try more conservative treatments such as drugs or physical therapy—what it found was that this widely used surgical procedure worked no better at relieving joint pain or improving function than a sham operation—in other words, a placebo intervention in which the surgeon merely pretended to operate.

And this was a groundbreaking study. It received a tremendous amount of media coverage. And the three of us are social scientists were not physicians, but we were startled by the study. And we began asking very basic questions. How is it possible that a widely used surgery wouldn’t work any better than a fake operation? And so what we began to do was investigate the medical literature. And what we found was this surgery had diffused into widespread practice in advance of any rigorous evidence that it really worked. Essentially, a number of surgeons began performing this procedure and they found their patients said that they felt better, but there really wasn’t rigorous data.

And that’s not so surprising because it turns out medical procedures, such as the kinds of things that your internist might do for you when you’re ill, or even surgeries, often become widely used without rigorous trials. We have a Food and Drug Administration that looks at the efficacy of pharmaceuticals, but there’s no FDA for surgery, for example. And so this turns out to be actually a very, very wide problem.

And what we learned as we started digging into the case is that this was really illustrative of a much broader problem in health care. Indeed, some experts believe that less than half of the medical care that Americans receive is based on adequate scientific evidence about what works best. Many treatment options have never been compared head-to-head with alternatives. So, for example, let’s say you have a bad back and you know there are different ways to treat it. You could try a drug, you could try physical therapy, you could get spinal fusion surgery. What patients would like to know is which of these treatments is really going to be best for me. And often the answer is we just don’t know. The studies haven’t been done. We just lack that information.

Steven Cherry You say that people in one part of the country get four times as many hip replacements as those in another and not for any sound medical reason. So why are there these regional variations in treatments?

Eric Patashnik Yeah, so basically there’s no centralized process in the United States to investigate alternative treatments for common medical conditions and determining what works best. And scholars at the Dartmouth Institute have found that there are remarkable variations in practice. In other words, patients with the same medical condition might receive very, very different treatment options and that these variances in how patients with the same condition are treated are not driven by, for example, regional differences in disease or patient preferences or even clinical evidence.

It might just be, for example, that physicians in Cleveland use a particular drug, whereas those in Boston use a different one. And we don’t ever have a system to figure out expeditiously which one is best for patients. And as a result of that, it could be the case that some patients in one part of the country are receiving inferior treatments and we might have no way of discovering that in short order. That’s, I think, a major problem that a lot of patients don’t realize. Even if you see a doctor that you trust, even if you have a very good relationship with your physician, the treatment that you may be receiving might be quite different from other patients in other parts of the country that have the very same condition. There was, I think, a recognition that this was a big problem. As part of the Affordable Care Act, there was an agency created the Patient-Centered Outcomes Research Institute [PCORI], whose mission is to fund and disseminate research findings on the comparative effectiveness of different treatments and diagnostic tests and other options. And I think that’s a very good thing. But we’ve only made a limited amount of progress in generating the information we need to answer those basic questions. 

Steven Cherry So to what extent is the problem that true evidence, especially via a large randomized double-blind controlled trials, is hard to come by? It’s expensive to do. It’s time-consuming. It’s rarely done. And if and when it’s done, it’s hardly ever replicated.

Eric Patashnik It’s true. There are there are major barriers and these studies can be expensive. But the information that comes from these comparative effectiveness studies is really a public good. It benefits all patients and payers and all stakeholders in the political system. So that’s one of the justifications for having a government role in subsidizing the production of these studies, because all of us would be better off if we had reliable information about what treatments work best and for whom and under what conditions. Individual physicians really don’t have a strong incentive to fund these studies or ability to fund the studies themselves. In fact, in the knee surgery case that I mentioned earlier, if it were not for the entrepreneurial initiative of a few physicians out of Texas, we might still not have those groundbreaking studies. It was really just because a few leaders decided that they really wanted to answer the question of whether this knee surgery worked. There was no system in place to ensure that those research holes are filled.

Steven Cherry Under our system, trials and other tests of efficacy are often in the hands of the large drug and medical device manufacturers. To the layperson, that sounds like putting the wolf in charge of determining the best way to build the chicken coop.

Eric Patashnik We do have a Food and Drug Administration that does a lot of tremendous work and they’re very valuable. And that’s also a benefit for the pharmaceutical industry. They, too, have an incentive to make sure that products that are sold to the public are seen as effective and trustworthy. So the industry itself does want a certain level of regulatory scrutiny. But the current FDA process, I think, while extremely valuable, has some limits. It’s often the case that drug companies are not required to study whether a new drug works better than, say, a cheaper alternative way of treating the same disease. If there’s medical and nonmedical options for treating a medical condition, say, for example, a surgery or a pharmaceutical agent, we have no process in place to ensure that those two options are forced to compete head-to-head. And so the FDA process, I think, is extremely valuable. And it does ensure that we find that drugs work better than a placebo, but we often don’t have what we really want to know, what patients want to know, which is which drug is best for me or should I be taking any drug.

We’ve also seen, I think over time, in part under industry pressure, a bit of a lowering of the evidentiary bar and even how we evaluate drugs. So oftentimes what the FDA is looking at is not answering the question that patients most want to know, which is, will this drug help me live longer? Will it help me improve my quality of life? Oftentimes, studies are looking at surrogate endpoints. For example, if I take this drug, will it lower my blood pressure or will it improve my cholesterol? And those surrogate outcomes, we hope, are correlated with the health outcomes that we really care about. But oftentimes the statistical causal relationship between them is much weaker than we would like it to be. And so sometimes we even have studies that are well done. But the questions that they’re answering are really not the ones that patients care most about.

Steven Cherry You would think that insurance companies would refuse to pay for a sham procedure or for a procedure that’s done four times in one region than another.

Eric Patashnik The insurance companies are in a difficult position because, of course, they would like to figure out what works best and conserve resources and not allocate them to low-value medical services. But in the United States, it’s very difficult for private insurers not to cover a medical treatment, for example, that is covered by the Medicare program. And the Medicare program really only looks at whether interventions are reasonable as judged essentially by physicians. The Medicare agency doesn’t really have the authority to examine whether a particular intervention is cost-effective or good value for money. And it’s also the case that individual insurers don’t have the resources to pay for the studies themselves, because, after all, if one insurer would spend hundreds of millions of dollars to answer the question, does this treatment A work better than treatment B? Well, once that study is done, the information would be available to other insurance companies, to their competitors. And so they don’t really have—the first insurance company doesn’t really have—an incentive to make that initial investment.

So, yes, payers would like this information, but individual insurance companies really don’t have a strong economic incentive to fund the studies. And it’s also difficult for them to deny coverage to treatments if the Medicare agency funds it.

The entire Medicare program, which was established in 1965, was, of course very controversial at the start. Many physicians and the AMA [American Medical Association] originally opposed Medicare. They were very concerned that government would be intruding on their clinical autonomy and second-guessing them. In exchange for essentially they’re buy-in, the government essentially said, look, we’re going to pay for health care for senior citizens, but we will defer to your professional judgment about the best way to treat patients. That’s not really government’s role. And, of course, Medicare has changed dramatically over the last decades. And I don’t want to say that Medicare doesn’t scrutinize treatments at all or make coverage decisions. It does. But that basic model of essentially deferring judgment or deferring decisions about coverage to physicians basically has remained the same.

And so we rely on physicians to exercise their best judgment to determine appropriate care. In many cases, that works very well. But if we have, for example, a breakdown of professional authority, if we have physicians in a particular practice area are widely using a surgical intervention, that doesn’t work, we don’t really have a good way of fixing that so quickly. And what we saw in that knee surgery case that I mentioned earlier was after that landmark New England Journal of Medicine study came out—and this is what was so disconcerting—a lot of the orthopedic surgeons that perform that surgery did not embrace it at all. They reacted extremely defensively. They attacked the study authors. They they made all sorts of arguments about what was wrong with the study.

And, of course, any study could have some flaws. And certainly every any study should be replicated. And we might not want to change practices based on a single study. But the overall behavior of the orthopedic community was one of essentially trying to push away evidence as opposed to embracing it as a way of learning what is best to treat their patients. In our research, we found that unfortunately, that practice has repeated itself over and over again.

There was another recent study just a couple of years ago called Orbita, which looked at millions of heart patients who had clogged arteries and they were receiving a stent inserted to reduce their chest pain. And this is another example of a very widely medical procedure. It’s expensive. It carries risks. It’s become the standard of care. And yet it’s diffused widely into practice on the basis of really little hard data. And the Orbita study was like the knee surgery finally done and it was a sham-controlled or a placebo-controlled trial in which some patients were randomized to receive a stent and others received no intervention at all beyond everyone in the trial receiving basic pharmaceutical drugs for cardiac disease. And what it found was that the patients who received a stent experienced no more improvement in chest pain or exercise tolerance—that was the other endpoint of the study—compared to patients who received a placebo procedure. And interventional cardiologists really saw that study as an attack on their specialty. And they lashed out at the trial and at the investigators. And it was another similar reaction of self-protective and defensive behavior of physicians that were not embracing the best available medical evidence.

Steven Cherry The book argues that the widespread disregard for scientific evidence or the lack of it is systematic within the medical community but it’s also political—that there’s little incentive for politicians to step in and demand medicine be more evidence-based. Is this physician pushback one of the reasons?

Eric Patashnik Yeah, I mean, I think what we’re talking about then is, essentially, as a society, we have a social contract. We delegate our authority to physicians and we give them the privilege of licensure and they can control who is a physician. They earn high salaries. They quite rightly are treated with great respect. And I should say, we think physicians are amazing people and we see our work as trying to help physicians for sure. But the way that social contract works is we rely on the medical community to self-regulate, to figure out if physicians are not practicing appropriate care, and to learn what would be best for patients.

And if that social contract is not working well, what we find in the book is it’s extremely difficult for government to do anything about it. Well, why is that? The main reason is when it comes to medical care, the public really trusts physicians. We did a bunch of public opinion surveys to try to understand how the public thinks about health care. And we found the bottom line of really scores of surveys that we did was when it comes to medicine, really the only actors that the American public trust are physicians. They don’t trust pharmaceutical firms. They don’t trust insurance companies, and they certainly don’t trust government.

Steven Cherry I mean, part of that trust is grounded in the fact that medicine is pretty arcane, complex, technical ... People go to school for years and years and years and train in various ways. I mean, if my life depended on my having an opinion on which version of string theory was more likely to be true, I’d study up on string theory, but I wouldn’t get very far. And I think neither would most people.

Eric Patashnik Absolutely. It’s completely understandable and rational for ordinary Americans to look to doctors for advice about what works best. I certainly do in my own life and would steer people away from just Googling medical conditions at random. And you can find a lot of misleading information on the Web. But what we really do want at the end of the day is the best available evidence about what treatments work best. And then that is a sort of systematic evidence base, and then ideally physicians would be using that kind of data and then taking into account the specific medical conditions and backgrounds and preferences of individual patients. So we certainly think there’s a major role for clinical judgment in medicine. Medicine is always going to be both an art and a science. What we argue in the book is that the ratio of art to science has gotten out of whack, and we’re relying too little on rigorous evidence and too much on idiosyncratic decision making.

Steven Cherry So I take it your solution would involve an FDA for surgery and other medical procedures.

Eric Patashnik Well we certainly think that there should be much more rigorous scrutiny of procedures.

That’s really a big part of medicine. We would like to see PCORI, the Patient Centered Outcomes Research Institute, begin taking on some of those harder questions in medicine. And then we would also like to see, I think, the Medicare agency begin looking more rigorously at the added value of particular treatments.

And if there’s a treatment that is extraordinarily expensive and there’s no evidence that it works better than a cheaper alternative, we don’t necessarily think that patients shouldn’t get it. They should certainly be free to choose it. But perhaps, for example, the Medicare agency should only pay in reimbursement up to the amount of the cheaper, equally effective treatment. So we really need, I think, more tailored kinds of coverage and reimbursement policies, as well as more rigorous studies. It’s been difficult, however, to move the needle on these kinds of policy solutions. The agency that I mentioned, PCORI, was very, very cautious in its first decade. It was just reauthorized, which I think is terrific, but it was very controversial. When it was first created back in 2010, it was seen as a rationing agency. It got caught up in charges of death panels, along with a lot of controversy, as part of the ACA.

Even in the current crisis, we’ve lacked the kind of rigorous information we need to figure out, for example, what covid therapeutics would be most effective. There’s been a paucity of randomized control trials on those kinds of questions. Of course, we’ve been in the middle of a pandemic. It’s quite understandable that physicians have been trying to do the best they can with available therapies. But other countries like the UK have done better in getting rigorous studies going during the pandemic so we can get quick answers to these life and death questions.

Steven Cherry So fundamentally, these are data questions and data scientists are reinventing that sort of thing. Do you think that with electronic medical records and deep learning studies of procedures and outcomes and so forth, that even without a new federal agency, we could get better data just from the data that we have and don’t use properly?

Eric Patashnik I think that’s a fantastic question. My colleague Alan Gerber and I, the coauthor of the book, as I mentioned, there was that landmark Orbita study about the efficacy of heart stents. And we had a conference a couple of years ago at Yale where we brought the lead author of the study, along with other leading cardiologists and social scientists together to look at what had happened in that case and why we are so often struggling with questions of data. And one of the things that I think was most exciting about the conversation is there really is an opportunity, I believe, to connect medical researchers and data scientists and other kinds of scholars to figure out how we can tease out rigorous causal inferences about what treatments work best from observational data.

Now, there’s a long history in medicine of reaching conclusions on the basis of observational data and non-randomized controlled trials, where it turned out that some of our conclusions were wrong.  For example, beliefs that hormone replacement therapy would reduce heart disease, or that aggressive treatments for breast cancer would be better than standard treatments of breast cancer. And it turned out that in both of those cases, that was incorrect after randomized controlled trials were done.

So for very good reason, I think some of the leading evidence-based medicine ... physicians have been skeptical about learning through methods other than RCTs [randomized clinical trials]. But in recent years, I think there really have been some breakthroughs in data science and other kinds of. Techniques that do allow us to learn what kinds of treatments work best. I’m hopeful that that kind of partnership in the coming decades will accelerate our ability to learn. We’re certainly going to need more RCTs; we’re doing too few of them. But I think there are other methods of learning.

The COVID pandemic is going to provide a remarkable opportunity to learn about the efficacy of a wide range of treatments, because we kind of had a national experiment, particularly during the early part of the pandemic during March and April. Basically, Americans and people around the world just stopped going to the doctor for all sorts of conditions. And of course, some of that was problematic. There were people that had serious medical conditions. They were fearful of going to the emergency room. But there were also people who might have had knee pain and they otherwise would have gotten the orthopedic surgery. There were people who had a sore back and they put off a procedure or they didn’t get a colonoscopy or they didn’t get some kind of screening for another medical condition. And what we don’t know is what the health outcomes were of this dramatic change in people’s consumption of medical services during this period. We have a remarkable opportunity now to figure out which of those medical interventions really were necessary and that the fact that people didn’t get them or got them in significantly fewer quantities was actually really bad for people’s health. And that’s going to be crucial to learn. So we can make sure that people do get those vital treatments and which are those services turned out upon reflection, actually weren’t so necessary. And that even though people skip some of those things, they didn’t turn out to have any negative health outcomes at all or perhaps even escaped a cascade of further unnecessary treatments or overutilization. Certainly, nobody wishes that this had happened. But now that it did happen and we had this once in a century kind of dramatic shift in people’s consumption of health care services, we really need to do our best to learn from this experience and figure out what the consequences were.

Steven Cherry Well, Eric, I started this podcast with a Bible verse, and I’m going to switch from the early Christians to the ancient Greeks. You might feel like you’re pushing a rock up a hill like Sisyphus, but like the similarly punished Prometheus, it was in the cause of empathy and imparting knowledge. So I thank you for co-authoring the book and thanks for joining us today.

Eric Patashnik Thank you so much. It’s a real pleasure to be here.

Steven Cherry We’ve been speaking with Brown University professor Eric Patashnik about the thesis of his co-authored book, Unhealthy Politics: The Battle Over Evidence-Based Medicine: How partisanship, polarization, and medical authority stand in the way of evidence-based medicine.

Radio Spectrum is brought to you by IEEE Spectrum, the member magazine of the Institute of Electrical and Electronic Engineers, a professional organization dedicated to advancing technology for the benefit of humanity.

This interview was recorded December 16, 2020 via Zoom using Adobe Audition. Our theme music is by Chad Crouch.

You can subscribe to Radio Spectrum on the Spectrum website, spectrum.ieee.org, or on Spotify, Apple, Google—wherever you get your podcasts. We welcome your feedback on the web or in social media.

For Radio Spectrum, I’m Steven Cherry.

Note: Transcripts are created for the convenience of our readers and listeners. The authoritative record of IEEE Spectrum’s audio programming is the audio version.

We welcome your comments on Twitter (@RadioSpectrum1 and @IEEESpectrum) and Facebook.

See also

Proof and Consequences

A new book explores the deceptive power of numbers
A conversation with Charles Seife (06 October 2010)