Quantitative Approaches to Stress-Testing Scientific Claims in Health Care

Analysis Group affiliate Dr. Christopher Worsham is a health care policy researcher on the faculty of Harvard Medical School and a practicing pulmonologist and critical care physician at Massachusetts General Hospital.

In his work, he applies traditional epidemiologic and advanced econometric methods to analyze large databases to better understand patient and physician behavior, resource utilization, and potential causal relationships within the health care system to inform health care policy.

In his research, Dr. Worsham investigates challenges of applying evidence-based medicine at the bedside, frequently focusing on what causal inferences can – and can’t – be drawn from real-world observational data.

Analysis Group Principal Brian Ellman and Managing Principal Jee-Yeon Lehmann spoke with Dr. Worsham about his approach to validating and analyzing observational data for use in both research and litigation.

What are some of the challenges that you face in applying quantitative methods to evaluate scientific questions in health care?

Randomized controlled trials are the gold standard for establishing cause and effect. Randomization of study subjects into a treatment or control arm of a study helps ensure that characteristics other than the exposure of interest are evenly distributed between the two groups. If randomization is done correctly, we can attribute any differences in outcomes to the treatment itself because we know the groups are similar to one another except for the exposure of interest.

The problem is, answering research questions using randomized controlled trials is difficult and expensive, and oftentimes not logistically or ethically feasible. As a result, many published biomedical research studies rely on observational data, such as those from electronic medical records or insurance claims databases.

However, relying on observational data can be prone to bias and yield spurious results if the data and/or methodologies used in the analysis are deficient. For example, if an analysis does not properly control for differences between groups of patients in the study or other factors that can bias or confound the measured relationship between an exposure and an outcome of interest, observed differences between comparison groups of patients will not reflect a true association between exposure and outcome and cannot be used to infer a causal relationship.

How can you minimize problems associated with bias?

It all boils down to the quality of the data and the methodology. Researchers can seek to account for confounding factors to minimize bias in their analysis, but this is a very difficult – and often impossible – task. These factors must be measured accurately, consistently, and reliably to be able to account for them in any analysis of observational data.

The biggest challenge, however, is that no matter how many high-quality measurements you collect, there are always other potential confounding factors that can be missed, both known and unknown.

Since it’s not always feasible to conduct randomized controlled trials, are there ways you can evaluate health care questions using real-world data and achieve the benefits of randomized controlled trials?

Yes! One way is to identify circumstances where patients have been randomized to one path or another accidentally – not because of a researcher’s coin flip. When this happens it creates what’s called a “natural experiment,” the product of random circumstances creating experiments in the real world.  

These natural experiments are happening all the time – we just need to be able to identify them in the data. We can study them using a variety of different empirical methods, ranging from traditional epidemiological tools to advanced methods of causal inference and econometrics.

In my book coauthored with Dr. Anupam Jena, Random Acts of Medicine, we dive into lessons learned from natural experiments in medicine conducted by ourselves and others using real-world data.

What is an example of a natural experiment?

Here’s an example (which we discuss in more depth in Random Acts of Medicine) which may be intuitive to anyone who has kids. My son happened to have been born in the summer; when we go in for his annual physical timed around his birthday, we’re always told to make a separate appointment in a few weeks to get his flu shot when it becomes available, typically in September or October. Other kids happen to have been born in the fall, which means they can get their flu shot at their annual check-up without having to schedule another appointment. In other words, because of the month in which they happened to be born, summer-born kids have a harder time getting a flu shot than fall-born kids.

 

Since birthdays are as good as random when it comes to the flu, kids are effectively randomized to either having to come back for a flu shot or getting it at their annual visit – giving us the opportunity to study the effect of this logistical barrier to vaccination (coming back for an extra appointment) while minimizing bias.

In fact, in a study published a few years ago in The New England Journal of Medicine, we found that kids born in the fall were about 10 to 15 percentage points more likely to get their flu shot than kids born in the summer. We also found that summer-born kids were more likely to get the flu and spread it to older family members.

This was an example where one could not ethically conduct a study by intentionally making it harder for kids to get flu shots, but we were able to study the natural experiment that occurred when it happened to kids, by accident, based on their birthday.

Why is this important for litigation?

From my vantage point of a practicing physician and health care policy researcher, an allegation in a legal case can be viewed as a hypothesis to be tested. Making the most of the available data by applying reliable epidemiological, biostatistical, and econometric methods when appropriate can yield valuable insights into the realities of a case, especially if there is an opportunity to take advantage of naturally occurring randomization.

The more an analysis can minimize potential bias, the more confident and precise we can be in drawing conclusions from a hypothesis test about possible causal relationships – or lack thereof – which are often critical questions in the case. Understanding how to minimize bias in observational data often is a key consideration in the reliability of evidence put forward in a litigation regarding potential causal relationships. ■