‘Everyone gets it wrong sometimes’ is a bad argument when assessing track records

Why there’s wrong and there’s wrong

There’s recently been a lot of coverage of the incoming US administration, and nominations of candidates who are underqualified – or in some cases even antiqualified – for major health roles. When their past exploits have (rightly) been scrutinised in public, a common counterargument from their supporters is ‘everyone gets it wrong sometimes’.

Take COVID. We shouldn’t be too critical about nominees’ pandemic views, goes the logic, because lots of people ended up saying things during the pandemic that didn’t age well. But I think this argument is unhelpful and unsophisticated. It’s a variant of ‘poisoning the well’, as I’ve written about previously. Rather than encourage proper evaluation of relative competence and skill, it instead flattens the playing field to the point of making evidence irrelevant. Everyone gets it wrong sometimes, runs the argument, so everyone is the same, so there’s no point trying to weigh up past behaviour or actions.

But evidence does matter. Because not all errors are the same. When it comes to getting things wrong in science and health, there’s wrong and there’s wrong. So, how can we tell the difference? I think there’s a few important criteria to consider.

How easy was it to not be wrong?

During COVID, one widely publicised claim was that the fatality risk was as low as 0.01% in countries like the UK and US (i.e. 1 in 10000 infections were fatal). But even a basic sense check showed this was nonsense. Over 50,000 people died of COVID in 2020 (and this conclusion holds regardless of the exact definition of death used), so for the fatality risk to be that low, over 500 million people would have needed to have been infected in the UK. Which is considerably larger than the UK population.

False claims that are easily disproved are arguably worse than ones that are more nuanced; it shows someone hasn’t even made a minimal effort to check their reasoning.

In other cases, a claim that is incorrect now would not have been so clearly wrong at the time. Take the role of COVID vaccines in reducing transmission. Because vaccines have not stopped variants like Omicron circulating, and the original clinical trials focused on prevention of symptomatic disease rather than transmission, some have suggested vaccination messaging should never have implied population benefits. ‘Get vaccinated for others’ was always a lie’, as one Dutch European Parliament member tweeted in 2022.

Yet vaccines were very effective at reducing infection – and hence transmission – of the original variant and Alpha in 2020 and early 2021. In fact, countries probably could probably have eliminated Alpha with high enough vaccine coverage. Analysis of UK contact tracing data suggests that vaccines even cut transmission of the Delta variant by around 50% on average.

Of course, we now know the Omicron variant emerged in late 2021, and the continued process of evolution (which I wrote more about here) means vaccines are no longer able to prevent waves of infection, even in theory. But based on the evidence available in early 2021, the claim that vaccines were substantially reducing transmission was not wrong, even if it would be for later variants. What’s more, concluding that the claim would not have held true in the future would have required some strong assumptions about the potential evolutionary trajectory of SARS-CoV-2.

How frequent was the error?

It’s not just about avoiding dodgy claims that are easy to sense check. It’s also about the relative frequency of those claims. If someone publishes 100 papers and 5 contain claims subsequently shown to be wrong, then – all things being equal – it’s a better track record than publishing only 5 papers and having them all make incorrect claims.

During COVID, I contributed to over 50 policy reports and dozens of scientific papers, while doing hundreds of media interviews, and posting hundreds of tweets. Even if the vast majority of my public statements had been totally correct, that would still mean several that haven’t stood the test of time (like this tweet about masks from Feb 2020).

How impactful was the error?

Not all statements are equal in terms of impact. A throwaway comment in a speculative Twitter post is very different from a coordinated media campaign advocating for a specific policy. So I think we also need to consider the health risks of a claim, i.e. the probability it is wrong multiplied by the impact it would have if wrong.

I found claims in mid-2020 that there wouldn’t be a second COVID wave particularly frustrating for this reason. Some of the loudest proponents had a strong PR team behind them, so they successfully skewed public debate towards flimsy speculation about whether a second wave would happen at all – and away from the much more important question of what to do about it.

What is the false positive vs false negative ratio?

In medical studies, we are traditionally four times more concerned about false positives than false negatives (specifically, studies are designed with 80% power, i.e. a 20% false negative risk, but use a 5% significance cutoff, i.e. 5% false positive risk).

If we take this as an indicative ratio of how to treat forms of being wrong, then we should be more critical of claims that say we know something when we don’t than claims we don’t know something when we do. In other words, if we evaluate public scientific and health claims like we do medical trials, overconfidence should be more penalised than underconfidence.

Avoiding a reputation for errors

There’s always a risk that a measure of competence becomes an incentive. Based on the above, perhaps people would just make fewer public claims, and keep these more ambiguous?

We should remember, however, that we’ve just considered errors so far. What also matters is getting things right, while also minimising mistakes. So a track record of 99 correct statements that are impactful, and a single ambiguous one, is better than 100 ambiguous statements that are useless.

When it comes to science and health, we should make statements that are as confident we think are justified, and likely to have positive impact. And we should accept that it is reasonable to be judged on the strength of this justification at the time we made the statement. Because not everyone is the same when it comes to getting things right and wrong, and if we’re going to evaluate people lined up for powerful positions, we need to know the difference.

Cover image: Elimende Inagella via Unsplash