The pop quizification of knowledge

Chasing the perfect snippet

“A troop.”

“Of how many persons?”

“Twenty men.”

“What sort of men?”

“Sixteen pioneers, four soldiers.”

“How far distant?”

“Five hundred paces.”

When Alexandre Dumas published the first serialisation of The Three Musketeers in 1844, he was paid by the line. The story goes that to make the most of the deal, he therefore added the character of Grimaud, a servant who spoke only in short, clipped sentences. Thanks to exchanges like the above, between Grimaud and the similarly verbally economical Athos, Dumas boosted the money he made per word.

Unfortunately for Dumas, the publisher eventually caught on, and declared that short lines didn’t count. In response, Dumas promptly cut Grimaud from the story.

The way we assess progress can sometimes incentivise people in unhelpful ways. ‘When a measure becomes a target, it ceases to be a good measure’ goes the famous adage from economist Charles Goodhart.

Like Dumas, careers can be built on measures that have become targets. In his book The Accountability Machine, Dan Davies points out that the outrageously lucrative academic publishing industry emerged as a useful way of judging academics. Getting papers in ‘good’ journals – where it is more likely to be read and cited – is often the target, rather than journals being measured as ‘good’ because they’ve published the best work.

As Davies puts it in the book:

The model persists because somewhere along the way, the journal industry managed to insert itself into the staff promotion and recruitment function of universities all over the world. In doing so, it created an extremely useful accountability sink for senior academics and managers of universities, while also solving an awkward and unpleasant interpersonal problem for them – how to judge the quality of scholarship without offending the scholars.

Chasing answers

A few months ago, OpenAI published a paper digging into why AI models hallucinate so readily. Their conclusion was that some of the blame lay with incentives:

Hallucinations persist partly because current evaluation methods set the wrong incentives. While evaluations themselves do not directly cause hallucinations, most evaluations measure model performance in a way that encourages guessing rather than honesty about uncertainty.

In other words, LLMs often behave like someone trying only to ace the final exam, rather than think deeply about a problem. This is not just a behavioural quirk of LLMs. There seems to be a growing attitude among humans that the point of reading is merely to be able to pass a pop quiz at the end. Why read a book when you could just read the key crib bullet points? A few years ago, that might have meant using one of those summarisation apps that extract insights (but don’t pay authors). Now it typically means asking an LLM (which is much the same thing).

In a recent piece, Sasha Chapin observed that conversations can be subject to similar efficiency constraints in the land of tech:

In the Bay, most gatherings have the sweaty air of Purpose. Discussions are held to uncover new information, not because it is good to be around each other. Conversations feel like podcasts and the hosts are not funny.

Knowledge is increasingly being redesigned around extractable outputs: test answers, summaries, bullet points. But this misses the point. Books and conversations aren’t just tokens to be processed efficiently by our eyes and ears. They are journeys in thinking and experiencing. Journeys that can bring the serendipity and struggle of deeper understanding.

If you’ve ever read a great novel, you’ll know what it feels like to not want it to end. If there’s a sadness at the final page, it’s because a powerful experience is over. Not because you now have to go and take a quiz.

---

Cover image: Ed Robertson

---