The other issue is confusing individual interventions and population interventions. For me, the great example is, would I be better off if I wore a bike helmet? And, would the population be better off if we legally mandated the wearing of bike helmets? It is common to see neurosurgeons being asked about their opinion on the second question, which is a long way outside their knowledge.
At one point in my public health career I was involved with "commissioning" - decisions about how much resource should go into different health specialties or services. I quickly realised that specialists all overestimated the prevalence of the conditions they treat. Not surprising, really - their clinics are full! But they mostly had a very handle on how common the condition actually was. Tertiary specialists - who were referred all the serious cases from a wide catchment area - were particularly susceptible to this...... Paediatricians, for example, lobbied hard (and, eventually, successfully) for the introduction of non-group-specific meningococci vaccine. About 10% of children who get ill with meningococcal disease die; and survivors are often severely affected, with eg amputation, deafness, or brain damage. It was very distressing to tell parents of the death or damage to their children. But the proportion of the population that gets I'll with meningococcal disease is very small. And, oddly, the economic "cost" to a society is quite limited; whereas rolling out a very expensive new vaccine across the entire young population is extremely expensive. With limited resources, could the limited pot of healthcare funding be spent better elsewhere? In the end it was the cumulative cost of treating damaged survivors for the rest of their lives that swung the decision. drugsincontext.com/vaccination-against-meningitis-b-is-it-worth-it ...... Similar examples are everywhere, whether you're looking at erectile dysfunction services (possibly less well funded than they should be because they're embarrassing), or the always-quote hip replacements...... Another factor is "service-led demand", or lack of demand. If services for a particular condition are so limited that there is no point in seeking a specialist appointment, GPs don't refer: so there is no demand, and therefore no perceived need to fund more of those services......
You make excellent points about population level vs. individual level analyses and claims. And about experts making big claims that might not be justified by the evidence, particularly outside of their own specialty. We see all of these frequently in the field of autism.
This reminds me of Tetlock’s finding that (academic) experts that talk to the press are worse forecasters than those who don’t. He doesn’t want to be on the show because he is uncomfortable answering questions outside of his narrow expertise? We have to have him!
It is all very well you castigating politicians making what you think are unsubstantiated claims, but this is no different from making this above statement without mentioning any of the huge caveats that apply to this Lancet study.
Basically the study was a retrospective calculation of what would have be the death toll if there was no vaccinations, but using an assumption of vaccine efficacy. The assumption of efficacy means the answer was baked into the method. I
notice you have praised the huge success of the initial Pfizer study, but this study was so flawed it should not have been published, and nearly all subsequent studies used similar methods of miscategorising all early cases as unvaccinated,( ie including those in the vaccinated group.) The data is so polluted by commercial interests and uncertainties, I think such categorical assertions made both by the press, politicians and 'establishment' scientists such as yourself should be suitably qualified
Thanks for your comment. You have previously argued on my other posts that there is no evidence for the efficacy or effectiveness of COVID vaccines, suggesting it could be anywhere between 0% and 100%. However, this overlooks a large body of evidence beyond the initial Pfizer trial. Most subsequent analyses have used population-level effectiveness data – based on real-world outcomes from large, independent cohorts – rather than just early trial efficacy.
It’s also incorrect to claim that 'nearly all' studies have ignored prior infection. Studies like SIREN, LEGACY, ONS, and REACT have explicitly accounted for prior infection status, and consistently found strong protective effects of vaccination. The reason researchers like me have confidence in the vaccine effect is not because of a single 2020 trial – but because multiple independent studies, across different settings and designs, have found very similar results.
On the Lancet study you mention - it does have a major limitation in that it only looks at direct effects, as I noted in the linked post. In reality, the indirect reduction in infection and infectiousness from vaccination (particularly against Alpha in 2021) would mean greater impact.
I want to keep discussion on Substack constructive, so if the same points are repeated without addressing the broader evidence base, I will need to limit further engagement.
Ioannidis' paper focuses particularly on small, exploratory, underpowered, biased, and non-randomized studies (of which there were many in the 2000s that were failing to replicate). The 'bias' term in his mathematical model (capturing things like post-hoc subgroup analysis and publication bias) is a particularly important driver of his conclusions. In contrast, for COVID vaccines, we had multiple large, pre-planned studies all addressing the same question (i.e. risk after vaccination), with high pre-study odds of an effect (e.g. based on experimental and early clinical data).
The BMJ paper you cite refers to a single trial with outcome switching and no independent replication, which is a very different situation to the hundreds of studies that have looked at the same COVID vaccine endpoints.
I'm not aware I've used the term 'settled science' anywhere (I generally refer to strength of evidence across studies), so this seems like an unrelated point.
The other issue is confusing individual interventions and population interventions. For me, the great example is, would I be better off if I wore a bike helmet? And, would the population be better off if we legally mandated the wearing of bike helmets? It is common to see neurosurgeons being asked about their opinion on the second question, which is a long way outside their knowledge.
At one point in my public health career I was involved with "commissioning" - decisions about how much resource should go into different health specialties or services. I quickly realised that specialists all overestimated the prevalence of the conditions they treat. Not surprising, really - their clinics are full! But they mostly had a very handle on how common the condition actually was. Tertiary specialists - who were referred all the serious cases from a wide catchment area - were particularly susceptible to this...... Paediatricians, for example, lobbied hard (and, eventually, successfully) for the introduction of non-group-specific meningococci vaccine. About 10% of children who get ill with meningococcal disease die; and survivors are often severely affected, with eg amputation, deafness, or brain damage. It was very distressing to tell parents of the death or damage to their children. But the proportion of the population that gets I'll with meningococcal disease is very small. And, oddly, the economic "cost" to a society is quite limited; whereas rolling out a very expensive new vaccine across the entire young population is extremely expensive. With limited resources, could the limited pot of healthcare funding be spent better elsewhere? In the end it was the cumulative cost of treating damaged survivors for the rest of their lives that swung the decision. drugsincontext.com/vaccination-against-meningitis-b-is-it-worth-it ...... Similar examples are everywhere, whether you're looking at erectile dysfunction services (possibly less well funded than they should be because they're embarrassing), or the always-quote hip replacements...... Another factor is "service-led demand", or lack of demand. If services for a particular condition are so limited that there is no point in seeking a specialist appointment, GPs don't refer: so there is no demand, and therefore no perceived need to fund more of those services......
You make excellent points about population level vs. individual level analyses and claims. And about experts making big claims that might not be justified by the evidence, particularly outside of their own specialty. We see all of these frequently in the field of autism.
This reminds me of Tetlock’s finding that (academic) experts that talk to the press are worse forecasters than those who don’t. He doesn’t want to be on the show because he is uncomfortable answering questions outside of his narrow expertise? We have to have him!
"Last month, a new study estimated that COVID vaccines had directly saved around 1.6 million lives between late 2020 and March 2023"
I assume this is a cut and paste from a previous post and actually refers to the Lancet article of Sept 2024 (https://www.thelancet.com/journals/lanres/article/PIIS2213-2600(24)00179-6/fulltext).
It is all very well you castigating politicians making what you think are unsubstantiated claims, but this is no different from making this above statement without mentioning any of the huge caveats that apply to this Lancet study.
Basically the study was a retrospective calculation of what would have be the death toll if there was no vaccinations, but using an assumption of vaccine efficacy. The assumption of efficacy means the answer was baked into the method. I
notice you have praised the huge success of the initial Pfizer study, but this study was so flawed it should not have been published, and nearly all subsequent studies used similar methods of miscategorising all early cases as unvaccinated,( ie including those in the vaccinated group.) The data is so polluted by commercial interests and uncertainties, I think such categorical assertions made both by the press, politicians and 'establishment' scientists such as yourself should be suitably qualified
Thanks for your comment. You have previously argued on my other posts that there is no evidence for the efficacy or effectiveness of COVID vaccines, suggesting it could be anywhere between 0% and 100%. However, this overlooks a large body of evidence beyond the initial Pfizer trial. Most subsequent analyses have used population-level effectiveness data – based on real-world outcomes from large, independent cohorts – rather than just early trial efficacy.
It’s also incorrect to claim that 'nearly all' studies have ignored prior infection. Studies like SIREN, LEGACY, ONS, and REACT have explicitly accounted for prior infection status, and consistently found strong protective effects of vaccination. The reason researchers like me have confidence in the vaccine effect is not because of a single 2020 trial – but because multiple independent studies, across different settings and designs, have found very similar results.
On the Lancet study you mention - it does have a major limitation in that it only looks at direct effects, as I noted in the linked post. In reality, the indirect reduction in infection and infectiousness from vaccination (particularly against Alpha in 2021) would mean greater impact.
I want to keep discussion on Substack constructive, so if the same points are repeated without addressing the broader evidence base, I will need to limit further engagement.
Well obviously it is your prerogative to limit the discussion as you wish. I just like start from the position of Ionnides famous paper, that most published research is false (https://journals.plos.org/plosmedicine/article/file?id=10.1371/journal.pmed.0020124&type=printable) and for a good example is the extraordinary saga of the paroxetine papers (https://www.bmj.com/content/351/bmj.h4629. There should be much greater awareness that much of 'settled science' is as misleading as any politician
Ioannidis' paper focuses particularly on small, exploratory, underpowered, biased, and non-randomized studies (of which there were many in the 2000s that were failing to replicate). The 'bias' term in his mathematical model (capturing things like post-hoc subgroup analysis and publication bias) is a particularly important driver of his conclusions. In contrast, for COVID vaccines, we had multiple large, pre-planned studies all addressing the same question (i.e. risk after vaccination), with high pre-study odds of an effect (e.g. based on experimental and early clinical data).
The BMJ paper you cite refers to a single trial with outcome switching and no independent replication, which is a very different situation to the hundreds of studies that have looked at the same COVID vaccine endpoints.
I'm not aware I've used the term 'settled science' anywhere (I generally refer to strength of evidence across studies), so this seems like an unrelated point.