How much does X suppress Substack links?

I designed an experiment to find out

First, a bit of good news: I’m pleased to say that the FT have named Proof: The Uncertain Science of Certainty as one of their books of the year! Thanks to everyone who’s already picked up a copy and shared their thoughts – it’s great to see it sparking so many ideas and conversations.

Now, let’s get on to the question of social media…

---

There have been many frustrating changes with X/Twitter in recent years. One is apparent suppression of external links, which has helped incentivise the worst of the platform: downweighting of evidence, downweighting of credit and downweighting of nuance.

Last year, I posted about the striking difference in engagement between Substack links I posted on Bluesky and X. Per follower, I found engagement was 10x if not 100x larger on Bluesky. Since releasing my book earlier this year, I’ve also noticed some strangely subdued content on X, like the post below Mishal Husain kindly shared. She’s got 240k followers and yet this only got 3k views?

So it was interesting to see Substack CEO Chris Best recently suggest that X had dropped the policy of hindering certain links, and traffic to Substack from posts on X was resuming:

Of course, as with everything X-related, what you see isn’t necessarily what you get, and Best soon posted an update to clarify that the new browser was pre-loading the links before the user sees them, meaning the bump was probably an artefact. But then came another update, indicating that things weren’t quite so bad, but also weren’t quite so good:

Maybe this announcement means things have changed? After all, my Bluesky vs X experiment last year wasn’t really able to disentangle a lack of engagement from link suppression from a general abandonment of X as a platform.

Experiments in engagement

To investigate how much X might be suppressing links, I decided to design a quick experiment. However, there were two big obstacles:

1. I didn’t trust the view stats on X. Not only is the platform quite buggy these days, and potentially bot-ridden, it’s also just not, well, that trustworthy in general.

2. There are lots of reasons a particular post might get attention at a given point in time, simply by chance. And if I posted two posts with and without a link at a similar point in time, maybe the sequence would skew results.

To tackle the first problem, I decided to use likes as a metric; this at least captures the engaged subset of users who saw the post. I therefore posted the following: “Doing a quick experiment on link suppression. Like this X post if you see it.” This would also have the advantage of flagging some engagement to the X algorithm, and hence testing how well the post performed beyond the basic ‘following’ feed.

To tackle the second problem, I used what is known as a cross-over trial design (or what the statistician William Gosset called an ‘ABBA’ design). I’d post the above with a link then, 5 minutes later, without a link. Then 12 hours later, I’d flip the order. Then 12 hours after that, I’d flip the order again.

I decided to run three pairs of posts, because this should be enough to show up a pattern if X was suppressing Substack links as much as I suspected it might be.

Here’s what the results looked like:

If we take the ‘no link’ vs ‘link’ posts, we can run a t-test (thanks also to Gosset, aka Student) to get an estimate of the difference. The average percent drop was –24% (p=0.02). In other words, this quick experiment suggests that including a Substack link alongside your X post could mean around a quarter fewer views.

I also found that the raw view count (below in thousands) was reduced by a similar amount on average (i.e. –23%):

However, there was weaker evidence for the difference in a t-test (p=0.16); the counts were so erratic, it was hard to disentangle a genuine effect from random variation.

If I’m honest, it’s not as bad as I thought it might be. But it still seems like there’s some unnecessary friction in the system, which is reducing the visibility of external content. It was also striking that X only invited me to ‘promote’ the posts that didn’t have a link.

Anyway, I’d like to thank to everyone who took the time to engage with my little link experiment. Or at least, the 3/4 of you that the algorithm allowed to see those posts.