Skip to content
What the technology actually is
AI Worth KnowingWhat the technology actually is

When a first draft costs almost nothing the bottleneck moves to checking it

Generating plausible text became cheap and verifying it did not, and a great many institutions were quietly built on the assumption that those two costs were similar.

By Manish Trivedi3 min read

A CNC laser machine operating in a high-tech industrial exhibition setup with vibrant lighting.
Photograph by Peter Xie via Pexels
Editorial note. Independent reporting and analysis. Nothing here is sponsored or paid for. How we work.

Effort used to be a filter nobody had to design

A great deal of ordinary institutional life relied on the fact that producing something took time. A job application implied an hour. A grant proposal implied a week. A written complaint, a product review, a scientific manuscript and a legislative consultation response all carried an implicit cost that limited how many arrived.

That cost was never the point of any of these processes, but it was load-bearing. It kept volumes at levels that a review process could handle, and it acted as a rough signal of seriousness — not a good signal, but a free one that nobody had to justify or maintain.

The cost has fallen sharply for anything that consists of well-formed text. Whatever else follows from that, the filter is gone, and it was not replaced by anything, because nobody had ever designed it in the first place.

The two costs were never symmetrical

Generating a plausible paragraph and establishing whether it is correct are different kinds of labour, and only one of them got cheaper. Checking a claim means finding a source, reading it, and confirming it says what it was said to say — work that involves other people’s systems, other people’s time and no shortcut.

That asymmetry existed long before any of this technology. What changed is the ratio. When producing a claim and checking it took broadly comparable effort, the system self-regulated. When production becomes almost free and verification stays where it was, any process that depends on checking everything it receives is in trouble arithmetically.

This is the mechanism behind most of the strain currently reported across editorial, academic, recruitment and support functions. It is not that the output is bad. It is that there is more of it than the checking capacity was ever sized for.

Institutions are responding in predictable ways

The first response is usually a detector, and detectors for machine-generated text have not proven reliable enough to carry consequences. False accusations are a serious harm, error rates vary by writing style, and non-native writers have been disproportionately flagged in several evaluations. Building policy on top of an unreliable classifier tends to create a second problem next to the first.

The second response is to change what is asked for. Interviews return, live tasks replace take-home ones, sources must be provided rather than cited, and applications require something a person can only produce by having actually done the thing. This is more expensive to run and it does address the actual mechanism.

The third is to accept the volume and change the review to sampling, thresholds or reputation. That works where the cost of a missed error is tolerable and fails where it is not, which is roughly the distinction between a marketplace listing and a medical record.

The counter-case deserves a fair hearing

It would be easy to write this as pure loss, and that would be dishonest. The effort filter also excluded people with less time, less confidence, weaker command of the dominant language of an institution and no access to somebody who could help them write. Removing a barrier that never measured merit is not obviously a bad thing.

Plenty of the work that just got cheaper was administrative overhead nobody defended — form-filling, boilerplate, summaries of documents, first drafts of routine correspondence. If the technology does nothing else, reducing the volume of text that had to be written purely because a process demanded text is a real gain.

The honest summary is that a filter was removed, that the filter was crude and unfair, and that the processes downstream of it were nonetheless calibrated to its existence. Both halves of that are true simultaneously.

What is speculative here

Predictions about where this settles should be treated with suspicion, including the confident pessimistic ones. Similar arguments accompanied cheap printing, the photocopier, the word processor and the spreadsheet, and in each case some processes collapsed, others adapted, and new work appeared that nobody had forecast.

The plausible case for adaptation is that verification tooling improves too, that provenance and attestation get built into more workflows, and that institutions redesign around the new cost structure within a decade. The plausible case against is that verification is fundamentally harder to automate than generation, because it requires contact with the world rather than with a corpus.

Nobody knows which dominates. What can be said without speculation is that the cost ratio changed, that the change is large, and that any process assuming otherwise is currently running on an assumption that no longer holds.

Common questions

Are machine-generated text detectors any good?

Not reliable enough to justify a serious consequence for an individual. Published evaluations show meaningful false positive rates that vary with writing style, and short texts are especially hard. Some tools are useful as one weak signal among several; none should be treated as evidence on their own.

Does this mean written work is no longer a useful assessment?

It means unsupervised written work is a weaker signal of what someone can do unaided, which was always a shaky inference. Assessments that observe the process, require defence of the content, or test something the text cannot substitute for remain informative. That is a design problem rather than an impossibility.

Why is verification so much harder to automate?

Because it requires checking a claim against something outside the text, and most of the world is not in a database with an interface. Retrieval systems help where an authoritative source exists and is machine-readable. Where it does not, the bottleneck is contact with reality, and no amount of computation removes that.

In The Worldworkverificationinstitutionsvolume
Manish Trivedi
Consumer editor, AI Worth Knowing

Manish has written about how it works, in the world, limits & risks for most of the last decade and prefers a plain explanation to a clever one.