How much uncertainty is too much uncertainty?
And does it matter where it comes from?
Suppose you flip a coin. If it comes up heads, I give you £20. If it comes up tails, you give me £10.
Would you take this bet?
Things are undoubtedly in your favour: on average, you’d expect to gain £5 with each such bet (i.e. 0.5 x 20 - 0.5 x 10). I suspect that many of you would therefore take the wager.
But what if more was at stake? For example, suppose that if it comes up heads, I give you £200,000, and if it comes up tails, you give me £100,000.
It’s the same probability of a positive outcome, and the same ratio of payoffs, but this time you have a 50% chance of taking a serious financial hit. So even if you would have taken the first bet, you may well prefer to decline this one.
From bankroll to beer
When it comes to risk, a lot of focus falls on the probabilities involved. But just as important is bankroll management. What is at stake relative to the uncertainty we face?
In the case of a fair coin toss, our uncertainty about the outcome is all down to the randomness1 of how it will land. This is what statisticians call aleatoric uncertainty: we know there’s a 50% probability of each outcome, but that doesn’t mean we can predict the outcome with certainty.
In real life, though, we also have the problem of estimating the probabilities in the first place. Imagine I give you a coin and tell you it’s biased, but don’t tell you by how much. Now you don’t just have the aleatoric uncertainty about the outcome; you have epistemic uncertainty about the extent of bias.
How much uncertainty is too much? In the early 20th century, William Gosset (aka Student) pioneered statistical methods like the t-test for small sample sizes, which came in handy in his role as a brewer at Guinness. The company was scaling up production and needed to experiment to optimise their processes without compromising quality.
Gosset didn’t always have the luxury of big datasets. One of his early experiments had a mere two samples to work with. He was therefore pragmatic about how much uncertainty mattered in practice. As he put it, the degree of certainty we should aim for depends on the:
‘advantage to be gained by following the result of the experiment, compared with the increased cost of the new method, if any, and the cost of each experiment.’
Even in the modern era, many scientists are still tied to the idea that a p-value below 0.05 (i.e. the probability of observing a result at least that extreme purely by chance) is ‘significant’ statistical evidence. But significant isn’t the same as meaningful, which Gosset understood well. A tiny – and in practice, meaningless – change might produce a tiny p-value from a large dataset. And a potentially important change might produce a larger p-value in a small experiment.
Early in his career, Gosset realised that he was wrong to assume there would be a neat threshold for discovering what mattered:
‘When I first reported on the subject, I thought that perhaps there might be some degree of probability which is conventionally treated as sufficient in such work as ours and I advised that some outside authority should be consulted as to what certainty is required to aim at in large scale work.’
Gosset would come to think that the notion of a statistically significant cutoff ‘seems to me to be nearly valueless in itself’. He once described a p-value of 0.13 as a ‘fairly good fit’.
Coins and lotteries
Which brings us back to the coin toss. How much is too much to risk? In the 1950s, John Kelly Jr at Bell Labs derived a formula for maximising the long-term expected growth rate of capital in the face of uncertainty. It would become known as the Kelly Criterion. If p is the probability of success and b is the ratio of the payoff if you win, it suggested you should risk a proportion f of your available bankroll:
In our example above, p = 0.5 for a coin toss and b = 2, so we should be willing to risk f = 0.5–0.5/2 = 0.25 of our available bankroll on such a bet. If we bet more than a quarter, the Kelly criterion implies there’s too much risk of going bankrupt in the long run; if we bet less, we’re leaving too much advantage on the table.
In practice, of course, we may not know p exactly. For example, quantitative betting syndicates have to estimate the probability that a horse or team will win from available data. While researching The Perfect Bet, several people I interviewed emphasised the benefits of ‘fractional Kelly’ betting, which involves only risking a proportion of the amount suggested by the Kelly criterion. In other words, they were taking a formula designed for aleatoric uncertainty and adapting it for epistemic uncertainty.
Kelly everywhere
Once you start to think in these terms, you will begin to see examples all over the place. Take the following meme that has circulated recently:
What would you choose? It’s tempting to respond in a flippant, subjective way. But the Kelly criterion gives us a rational framework to analyse the choice.
In short, pushing the green button gives a 50% chance to win $100m versus losing $1m (i.e. the guaranteed amount you’ve given up2). So in Kelly terms, we have p=0.5 and b=99 (because your net profit would be $99m with the green button). Hence:
This implies it’s optimal to bet around half your bankroll in such a situation. So given a fixed $1m is at stake for a loss in this example, the Kelly criterion would suggest it’s only worth gambling on the green button if you already have more than $2.02m in the bank.
From beer to buttons, the above shows that risk taking is not just about raw probabilities. When it comes to real-life decisions, it’s also about how confident we are in those probabilities – and how much rests on the result.
If you’re interested in reading more about uncertainty, risk, Guinness and Gosset, you might like my latest book Proof: The Uncertain Science of Certainty.
Cover image: Eduardo Soares via Unsplash
Strictly speaking, a coin toss isn’t actually random - it follows the laws of physics. But the physics is so sensitive to small changes, like initial coin speed, that it’s effectively unpredictable and we can reasonably model it as a random 50/50 outcome.
Technically, this is an opportunity cost rather than a loss.
Thank you, Adam. It reminded me of another issue. Take an intervention that will certainly save much more money than it costs (say, early intervention for eating disorders). Should you invest in it? Of course! But what if you can't now afford the investment?
Adam, you always make us think! thank you. This concept you taught us today will be interesting to try in patient decision making; the notion of statistical significance, clinical significance and Kelly Criterion.