Why do people assume all models are forecasts?

Analysis informing a decision shouldn’t try and predict the decision it’s informing

Imagine you’re playing a game of poker, with a big pot of chips in the middle of the table, and you can’t decide whether to call or fold. To help your decision, what you probably want is an analysis of the implications of your possible choices, based on available information (i.e. a scenario model). This is why poker players often use calculations like pot odds, to decide if the chance of winning the hand justifies the risk of calling the bet.

What you don’t want in this situation is someone whispering in your ear ‘I reckon you’re going to fold’. In other words, you want a decision-support tool, not a forecast of your behaviour.

Now, let’s suppose an analysis of available information reveals that the odds are very much against you if you call, i.e. you’d expect to lose heavily. So you decide to fold. You wouldn’t then say ‘well the analysis was wrong, because it said I’d lose lots of money and I didn’t.’

So why do people often conflate things in the same way when it comes to disease scenario models? It’s striking how many people wrongly assume that epidemic model = forecast.

There is a (small) subset of disease dynamics research that is focused on forecasting, particularly for seasonal infections like influenza, which generally don’t trigger large reactive policy responses mid-outbreak¹. But more commonly, models are either used to estimate key characteristics of interest, or as decision-support tools.

Of the forty or so analytics reports I contributed to the UK SPI-M-O advisory group for COVID, hardly any involved a forecast. There were lots of predictions that could be tested as more data arrived, from estimates of severity, susceptibility and superspreading to transmissibility of novel variants and effectiveness of different control measures (e.g. rapid testing). As I’ve written about previously, such analysis always requires some form of model, to link the data we observe to the process or parameter we’re trying to estimate. But it wasn’t the same as saying ‘this is what I think the epidemic will do next.’

Probably the only clear forecast I led on was for Delta (known as B.1.617.2 at the time). In early May, we predicted that if the variant was genuinely as transmissible as we estimated, and the re-openining ‘roadmap’ continued as planned, then ‘the majority of sequenced cases in the UK would consist of this variant by mid-May 2021’:

Why is it so common to assume models are forecasts?

One of the most prominent, and frequent, modelling outputs we use in daily life are weather forecasts (especially if you live in the UK). Crucially, weather is an external influence on our lives. We don’t get to influence the outcome; carrying an umbrella because you reckon it will rain won’t change the chances of it raining.

The same is true of many events that statisticians first honed their prediction methods on a century ago. There was particular interest in gambling outcomes like roulette, which could be watched and analysed. Predicting red will come up half the time doesn’t change the odds.

But a disease epidemic is not a roulette table. Public and government actions can have consequences, from reductions in social contacts reducing transmission to vaccine coverage reducing susceptibility². Even so, there seems to be a temptation among some groups of commentators to treat all epidemics as something to be watched and predicted from afar, rather than a policy question that can be acted upon.

In some cases, this can lead to circular predictions about what might happen. If our forecast is that ‘people surely won’t let it get out of hand’, the model won’t predict a bad outcome because it assumes that a bad outcome will always be averted. To return to our poker example, it’s a bit like someone telling you ‘don’t worry, whatever you do will work out fine, because I’ve predicted you won’t choose to do anything that will lose lots of money.’

This is why it’s so important to distinguish scenarios – with clear assumptions – from forecasts. If a modelling analysis is informing decisions, it’s not that useful for the model to try and predict the decision it’s informing.

---

If you want to read more about data, models, decisions and policy, you might like my book Proof: The Uncertain Science of Certainty.

Cover image: Yeshi Kangrang

1. With the exception of some regional examples, like school closures in Hong Kong.

2. I outlined the best available evidence on impact of NPIs in this Boston Review piece, and Fig 5 here is a good illustration of the reduction in fatality risk following introduction of vaccination in the UK (alongside the many papers with data from trials and real-world effectiveness studies).