Skip to content
What the technology actually is
AI Worth KnowingWhat the technology actually is

Answering the question instead of listing the links unpicks the bargain the web ran on

Search engines took material and returned attention, and a system that synthesises an answer keeps the material without returning anything, which changes the economics that funded the sources.

By Manish Trivedi3 min read

Close-up of a modern, monochrome robot arm against a dark background. Futuristic technology concept.
Photograph by Pavel Danilyuk via Pexels
Editorial note. Independent reporting and analysis. Nothing here is sponsored or paid for. How we work.

There was an exchange, and it was mostly unwritten

For roughly three decades the arrangement worked like this. A search engine crawled a site, indexed what it found, and in return sent visitors to it. Nobody signed anything. The exchange was enforced by a convention allowing a site to ask not to be crawled, and by the fact that almost nobody wanted to be left out.

That flow of visitors is what paid for the material. Advertising, subscriptions, sales, reputation — whichever model a site used, arrival was the precondition. The index was not the product being monetised; it was a directory pointing at products that were.

The arrangement was never entirely comfortable. Summaries shown directly in results had already been reducing the need to click for years, and publishers had already been complaining about it. What has changed is the degree, not the principle.

Synthesis removes the destination rather than reordering it

A system that reads several sources and composes an answer has completed the transaction. The material was used and the visitor never left. Citations may be shown, and evidence about how often they are followed is mixed and mostly held by the parties least motivated to publish it.

For some queries this is straightforwardly good. Somebody wanting a unit conversion or a fact was never going to become a reader of the page that supplied it, and sending them there was inefficient for everybody. The kind of traffic being lost first is the kind with the least value to the site receiving it.

For other queries it is more serious. Material that took real work — investigation, testing, expertise — competes for attention on the same terms, and its distinguishing quality is not visible in a synthesised paragraph. The system draws on it precisely because it is good, and returns nothing that funds the next one.

The incentive to publish is what is actually at stake

If material is used without generating anything in return, the rational response over time is to publish less of it, publish it behind a barrier, or not gather it in the first place. That is not a moral claim about fairness; it is a straightforward description of what happens when a revenue mechanism is removed and nothing replaces it.

There is an awkward second-order problem here that both sides acknowledge. Systems answering questions depend on a supply of new material to be accurate about a changing world. A pipeline that degrades its own inputs is not stable, whatever anybody thinks about who deserves what.

Whether this actually plays out is uncertain and it is worth marking as speculation. Publishing has survived several predicted extinctions, substitution effects are notoriously hard to measure, and the counterfactual — what traffic would have done anyway — is unavailable.

The responses so far are partial and pull in different directions

Some sites have blocked the crawlers used for training, using the same convention that governed search crawling. Some have signed licensing agreements. Some have moved material behind registration or payment. Some have done nothing, on the reasonable ground that being absent from the answer is worse than being uncompensated in it.

That last calculation is the interesting one, because it is the same calculation that made the original bargain self-enforcing. A collective action problem in which nobody can afford to withdraw individually is a weak position from which to negotiate, and it is the position most publishers are in.

Litigation and regulation are proceeding in several jurisdictions with no consistent direction yet. Outcomes will vary by country, which suggests the eventual arrangement will be a patchwork rather than a settlement.

What a reader should take from this

The change is not a technology arriving and displacing an industry, which is the shape the story is usually told in. It is a specific exchange being renegotiated, where one side has considerably more information about what is happening than the other. The information asymmetry is arguably the more durable problem.

It’s also worth resisting the framing in which everything was fine before. The previous arrangement concentrated enormous power in the ranking of results, rewarded material optimised for that ranking rather than for readers, and hollowed out plenty of publishing on its own. Neither the current state nor the previous one is a good baseline.

The honest position is that a mechanism which funded a great deal of freely available material is weakening, that no replacement has established itself, and that predictions about where this lands are worth very little at present.

Common questions

Can a site prevent its material being used in generated answers?

It can signal a refusal through the same file that governs crawling, and compliance is voluntary and varies. A signal that depends on goodwill is a weak instrument, which is why the argument has moved towards contracts and courts.

Do citations in a generated answer solve the problem?

They help with verification and they do not restore the traffic, since the answer has already been given. They also raise a separate question about whether a cited source genuinely supports the sentence attached to it, which is not guaranteed.

Is this only a problem for news publishers?

No. Reference material, technical documentation, forums where people answer each other’s questions, and specialist hobbyist writing are all affected, and some of those had far weaker funding to begin with.

In The Worldsearchpublishingdeployment
Manish Trivedi
Consumer editor, AI Worth Knowing

Manish has written about how it works, in the world, limits & risks for most of the last decade and prefers a plain explanation to a clever one.