History
A simple pattern-matching program showed people would supply the understanding themselves
A conversational program written in the 1960s worked by reflecting sentences back with almost no analysis, and the reaction it provoked disturbed its author enough to redirect his career.
By Zoya Rahman3 min read
A program with no model of anything
In the mid-1960s, a researcher at an American university wrote a program that could conduct a written conversation. It worked by matching the user’s input against a set of patterns and transforming the matched text according to rules, so an input containing a phrase about family might produce a question about family, assembled from the words already supplied.
The best known configuration imitated a therapist of a school that favoured reflecting a patient’s statements back to them. This was an inspired choice, because that conversational style makes an absence of world knowledge look like professional technique rather than incapacity.
The program held no representation of the conversation, no knowledge of the subject, and no model of the person typing. When no pattern matched, it produced a generic prompt inviting the user to continue. That is the entire mechanism.
People engaged with it far more seriously than expected
The reaction is the historically important part. According to its author’s later account, users conversed with the program at length and with evident personal investment, some preferring privacy while doing so, despite understanding that they were typing at a piece of software.
Some who knew exactly how it worked still reported the experience as meaningful. That combination — full knowledge of the mechanism alongside a genuine sense of being heard — is what makes the episode more than an anecdote about naivety.
The tendency it revealed acquired a name and is still referred to: people attribute understanding to a system on the basis of surface conversational behaviour, and the attribution survives explanation of how the system works.
The author became one of the technology’s sharpest critics
Rather than treating the response as validation, its creator found it alarming, and spent much of the following decade arguing that certain applications of computing were inappropriate regardless of whether they were technically feasible. His central objection was not that machines could not perform particular tasks, but that some tasks should not be delegated to them even if they could.
His writing distinguished between deciding and choosing — between computation that produces an answer and judgement that carries responsibility — and argued the second cannot be handed over without losing something that matters. That framing remains one of the more useful contributions to the debate, and it is not primarily a technical one.
His position was contested at the time and remains so. Critics argued he underestimated what systems would eventually do and overstated the harm of using them in sensitive settings, and reasonable people continue to hold both views.
Why the effect works, mechanically
Conversation is enormously cooperative. Listeners fill gaps, infer intent, forgive irrelevance and construct coherence from fragments, because in ordinary life the other party is a person whose contributions merit that effort. These habits are automatic and they do not switch off when the other party is a program.
Reflecting a person’s own words back to them exploits this directly: the content is theirs, so it is guaranteed to be relevant, and the apparent understanding is their own understanding returned. Very little needs to be added for the exchange to feel responsive.
A further factor is that people tend to interpret text as evidence of a mind, because for the whole of history it was. Written language had exactly one source, and the inference was sound until quite recently.
What the episode does and does not tell us now
It provides an important caution: the impression that a system understands is unreliable evidence, since a program with no representation whatsoever produced that impression in people who knew better. Any argument about modern systems that rests primarily on conversational impression is standing on ground that was tested long ago and found weak.
It does not, however, show that current systems are merely elaborate versions of the same trick. That comparison is made frequently and it flattens a real difference: the older program manipulated the user’s own words by rule, while a modern model has been fitted to an enormous quantity of text and does generalise to inputs it has never seen.
The honest reading is narrower and more durable. Human judgement of understanding is not a reliable instrument, we have known this for a long time, and whatever the right way to assess these systems turns out to be, it is not how the conversation feels.
Common questions
Was the program intended to be a therapy tool?
No. It was written to explore natural language processing by pattern matching, and the therapeutic persona was chosen because it made the program’s lack of knowledge unobtrusive. Proposals to use such systems clinically emerged from others, and were among the things its author objected to most strongly.
Are modern systems just a bigger version of it?
Not in mechanism. The older program applied hand-written transformation rules to the user’s input, while modern systems are fitted to vast text collections and produce genuinely novel output. The continuity is in the human response rather than in the technology, and conflating the two makes both harder to think about.
Why does knowing how it works not break the effect?
Because the response is not a considered belief that can be corrected by information. It is an automatic social reflex operating on the surface features of the exchange, and reflexes of that kind persist alongside accurate knowledge, in much the way an optical illusion continues to work after it has been explained.
Deputy editor, AI Worth Knowing
Zoya joined to cover how it works, in the world, limits & risks and stayed for the awkward questions and would rather show the working than assert the conclusion.
