Epidemiology has a causality problem

If we think one thing is causing another, we should try and show it

I’ve seen a new paper shared quite a lot recently. It’s called ‘Effects of education on adult mortality’ and it concludes that, on average, there is about a 2% reduction in mortality per additional year of education.

It’s a striking finding, and I’ve noticed several people sharing the paper with a comment along the lines of ‘Education is so important, just look at how much it lowers risk of death’. This interpretation is perhaps unsurprising, given the institution behind the research announced it by stating: ‘Education is statistically proven to lead to a longer life.’

But does the paper really show that more education causes a reduction in risk of death? And hence, by implication, people who die early have an increased risk because they haven’t been educated as much?

No, I don’t think it does. The study just shows that education and mortality are inversely correlated (or ‘associated’ as epidemiologists like to say). In other words, when there is more education, there is less mortality on average.

If we see a correlation between two things, one possibility is that one thing is causing the other. Especially if the first thing (like education) comes before the second (like adult mortality):

But that’s not the only possibility. There might be another factor that is causing both more education and less mortality (e.g. a factor like wealth). In this case, less education wouldn’t cause higher mortality. The two would only appear to be correlated because of their relationship with this other factor:

The above paper accounted for some other factors that might influence individual mortality, like age, sex, and marital status¹. But there are still lots of other things that could influence both education and death. Hence the description of an ‘association’ in the paper, rather than a clear cause.

I’m not meaning to single out this particular study. The authors have put a lot of effort into collating and analysing the data, and – with careful interpretation – it does have the potential to add to our knowledge base about education and health. What’s more, traditional academic norms mean that journal editors and reviewers expect papers to report results in this ‘association’ based way. (Even if the authors later talk of things like the ‘protective effect’ of education in the discussion, while also saying ‘we are unable to make causal claims about the effect of education across the life course’). Many, many papers have done the same thing; this is just a recent and prominent example.

But I’ve increasingly come to realise how unhelpful these norms are. If we’re interested in whether something increases or reduces a health risk (which, ultimately, is the aim of much health research), we should frame our analysis around that cause-and-effect question. Of course, there may be some instances where we’re more interested in predicting health outcomes (e.g. for clinical prognosis, or to inform healthcare demand planning). And if this is our goal instead, we should say so.

Darren Dahly summed it up nicely in this recent piece:

People often say their goal is to identify “risk factors”. But what does that mean? Some people use the term to indicate potential causes of outcomes. Then just say cause. Others use it to identify predictors of outcomes. Then just say predict.

Getting closer to the cause

The problem with investigating the effect of education on later health is that we can’t easily run an experiment; we can’t randomly withhold education from some children but not others, then see what happens decades later. So instead we have to rely on the ‘observational’ data we can collect.

There are methods for investigating cause-and-effect using such data, by accumulating complementary information to get closer to what is really happening. Austin Bradford Hill’s classic paper ‘The environment and disease: association or causation?’ gives a good overview of what evidence we should consider here.

One of the pieces of evidence he mentions is probably the most common – if crude – approach to investigating causality from observational data. This involves looking for correlations and trying to adjust for any additional factors that could be influencing both things. If we adjust for everything correctly (which is very difficult, if not impossible, in reality) then any correlation that remains should reflect a true cause-and-effect relationship. Among other things, Bradford Hill also noted that causation is more likely if we see a ‘dose–response’ relationship, with more cause linked to more effect (as was observed in the above education paper, with mortality lower by 2% for each additional year of education).

But as Richard McElreath pointed out in this nice series of posts, if we’re interested in causality, there are now some much more thoughtful and targeted alternatives to this basic correlation-and-adjust method. We could use causal design to get at the cause-and-effect we care about, or define a full probability-based model of the process we’re investigating.

What we shouldn’t do is focus only on correlations if what we really care about is determining causes. If we want to work out if one thing causes another, we should design our analysis to tackle this problem, and make it clear how close we are to an answer. Despite academic traditions, we should also use language in a way that makes logical sense for the problem we’re talking about. If we’re making conclusions about ‘risk factors’, we should be confident that modifying them will change the outcome. If we talk about a factor having a ‘protective effect’, we should have evidence that removing this factor will reduce protection.

As Julia Rohrer put it in this useful primer:

Carefully crafted language will not prevent readers—let alone the public—from jumping to causal conclusions, and many studies that are based on observational data will probably get published only because they suggest that they are able to provide information about meaningful causal effects.

I don’t think it’s particularly informative to have vast numbers of epidemiological papers basically saying ‘this definitely isn’t in any way causal evidence, but here are some policy interventions we think are supported by this not-causal evidence’. Rather than dance around the issue of causality when dealing with causal questions, studies should tackle it directly. How much causal evidence was there before a new study was done? How much additional causal evidence does the new study provide? What more work is needed to be more confident about this potential cause-and-effect?

After all, if we don’t know if one thing is really affecting another, we can’t be sure that improving that thing will have any a real effect.

1. The authors also looked at the socio-demographic index of the country each study was conducted in, but this doesn’t account for variation in socio-demographic factors between individuals within a study.