Excess mortality or excessive assumptions?

How to make a popular metric tell any story you like

Which country escaped the worst of the COVID pandemic?

A recent article by the Directors of the US National Institutes of Health gave a confident answer. It claimed that Sweden ‘was the best in the world at protecting human life during the Covid pandemic. It had the lowest level of age-adjusted, all-cause excess deaths in the world between March 2020 and December 2024.’

Excess mortality is commonly quoted when comparing the impact of COVID in different countries. The metric is often reported as if hard data, but it’s actually the output of a scenario model – and it can depend on some very strong assumptions.

Let’s start with the graph the above article links to¹. In this plot, Sweden comes out with lowest cumulative excess mortality among the comparators considered, even above New Zealand by the end of 2024:

Notice anything unusual about the above plot? In particular, look at the sign of cumulative excess mortality: it is negative for both Sweden and New Zealand. It’s well established that New Zealand had a drop in mortality in 2020-21, given reduction in other seasonal infections. But is it really plausible that Sweden would have seen more mortality over the past six years had the COVID pandemic not happened?

Excess mortality and excessive assumptions

‘The death-rate is a fact; anything beyond this is an inference.’ –William Farr (1807-1883)

Mortality is something that can be measured. But excess mortality is a hypothetical comparison. It asks: how many more deaths occurred than expected? Which requires that we come up with some model of what ‘expected’ looks like.

To illustrate the challenge, here is Sweden mortality from 2010-19. What would we have expected to happen in 2020-24 without the COVID pandemic?

The excess mortality plot earlier used the average mortality in 2017-19 as the baseline for what would have been ‘expected’. But you’ll notice that 2018 and especially 2019 were quite a lot lower than 2017, so taking the average fixes the ‘expected’ baseline at quite a high level. Hence post-2020, this assumption leads to Sweden having a substantially negative excess mortality.

But is a fixed baseline really that plausible if there’s a downward trend in mortality prior to 2020? What if we instead fit a linear trend to 2017-19 and use that as the expectation? Under this assumption, the actual mortality in Sweden ended up much higher than the expected trend:

This subtle difference in assumptions about the baseline matters. If we use the first assumption, with a fixed baseline, then Sweden has very low – and negative – excess mortality, lower than nearby countries and New Zealand.

If we instead use a linear trend as the post-2020 baseline for all countries, Sweden ends up with a much higher – and positive – excess, with all the others lower:

Cherry-picking a ranking

Because excess mortality calculations can be so sensitive to the baseline assumption we choose, and we are effectively extrapolating from a short time series (2017-19) to a longer time period (2020-24), it’s possible to cherry-pick baselines that can generate all sorts of different rankings.

For example, if we pick 2018-19 for the linear trend baseline, then Sweden comes out worse than even the US:

In contrast, if we pick a (totally arbitrary and unjustified) 2013-15 period for the linear trend to extrapolate from, we could even make New Zealand come out with the highest excess mortality of the three, and the US with the lowest:

Getting the right trend

This doesn’t mean it’s impossible to say anything at all about the impact of the pandemic, and we should all just cherry-pick our favourite baseline. Regardless of the assumption we use, Sweden was hit hard by COVID in 2020, like many other countries in Europe. It’s the extrapolation beyond this point that becomes very sensitive to assumptions.

We can use statistical models to identify which trend assumption might be more or less plausible given the data. For example, if we fit a generalised additive model (i.e. a flexible extension of a linear regression), it suggests there is evidence for a near-linear decline as the most reasonable expectation:

However, that low 2019 data point makes it difficult to be confident. If we focus only on data from 2015-19 instead, we get a steeper downward trend, which would produce a higher estimate of excess mortality:

It’s somewhat ironic that the country in Europe that was an outlier in its COVID response also had an outlier mortality data point immediately before the pandemic, making it near impossible to reliably compare it to other countries.

This is a useful reminder that excess mortality is not data. It is a scenario model for a reality that didn’t happen – and it depends a lot on the assumptions we make along the way.

1. Here we’re taking the calculations on that website at face value, given this was cited as evidence for the comparison. The aim in this post is to focus on the claim and the concept, rather than the specific people behind them.