People attribute a mind to these systems reliably, and the design does not discourage it

The attribution happens before anybody decides to make it

People read intention into very little. A pair of dots and a curve is a face. A shape that changes direction is chasing another shape. This is not credulity; it is a fast perceptual habit that is useful nearly all the time, because for most of human history anything behaving purposefully was in fact purposeful.

Fluent language triggers it more strongly than almost anything else. Coherent sentences that respond to what you said, in your language, adjusting to your tone, are the signature of a mind in every previous instance any of us has encountered. The response is not a considered judgement that can be argued away.

This means the usual remedy — telling people it is not a person — has limited effect. Somebody can know perfectly well what they are using and still feel that they are being understood, because the knowing and the feeling are produced by different parts of the process.

Several design choices push in the same direction

First-person language is the largest. A system that says it thinks something, or that it is happy to help, is using the grammar of an inner life. Alternative phrasings exist and are more awkward to read, which is a real trade-off rather than an oversight.

Names, personalities and conversational manner all add to it. So does the pause and the word-by-word appearance of text, which reads as composition even though it reflects how the output is produced. Deference and agreement, which come from the training that shapes assistant behaviour, are received as courtesy.

None of these is deceptive on its own and the combination is powerful. There is a legitimate argument that a natural interface is simply better and that pretending otherwise makes systems harder to use. There is an equally legitimate argument that the interface is making a claim the technology cannot support.

The vocabulary of the field carries the same freight

The technical terms were borrowed from psychology before anybody worried about what they would imply outside the laboratory. Learning, attention, hallucination, memory, understanding, reasoning: each names a mechanism that resembles the human phenomenon in some respect and differs in most others.

Inside the field this is harmless shorthand, because everybody knows what the mechanism is. Outside it, the words do exactly what words do, and a reader encountering them has no way to know which parts of the ordinary meaning were meant to travel. The term for confident fabrication is a particularly unfortunate example, since it implies a perceptual error rather than the system operating normally.

Renaming things at this stage is largely impractical, since vocabulary settles by use. What remains available is being explicit about what the words are doing, which is a poor substitute and better than nothing.

The consequences are practical, not just philosophical

Attributing a mind changes how output is treated. A stated reason is received as an account of what happened rather than as generated text. Confidence is read as knowledge. A refusal is read as a decision by somebody, which is why people argue with these systems and sometimes succeed, in the sense that persistence changes the output.

It also affects disclosure. People tell a system that seems to be listening things they would not type into a form, which has consequences for what ends up stored and where. The interface encourages a register of confiding that the underlying arrangement does not warrant.

And it affects reliance. A tool that feels like a colleague is checked less than a tool that feels like a spreadsheet formula, and the checking is the part that was carrying the safety.

Emotional attachment is a real phenomenon and the evidence is thin

People form attachments to conversational systems, sometimes strong ones, and this was documented with far cruder programs decades ago. Reports of it now are numerous, and rigorous evidence about how common it is and what it does to people over time is limited, because the question is difficult to study and the technology is recent.

Two positions are defensible. One holds that this is straightforwardly harmful, exploiting a perceptual vulnerability for engagement. The other holds that people find genuine value in it, that loneliness is a real problem, and that dismissing the value on the grounds that the other party is not conscious is an argument about metaphysics rather than about wellbeing.

Both should be held provisionally. Anybody stating confidently what the long-term effects will be is not working from evidence, because the evidence does not yet exist.

The useful stance is neither belief nor performance

Attempting to purge every mentalistic word makes explanation almost impossible, because there is no other vocabulary for describing behaviour at this level. Saying that a model appears to weight one factor above another is far clearer than the mechanical alternative, and it is not a claim about experience.

The workable discipline is to notice when the metaphor is doing load-bearing work in an argument. If a conclusion depends on the system wanting, knowing or intending something, the conclusion needs restating in mechanical terms before it can be trusted. Usually it survives, sometimes in a weaker form, and occasionally it does not survive at all.

That’s a habit rather than a rule, and it is the only defence that has held up. The perceptual response is not going away, and neither is the interface that provokes it.