Limits & Risks
A system’s account of its own reasoning is a reconstruction, not a record
These models can be asked why they answered as they did and will produce a fluent explanation, which is generated by the same process that generated the answer and carries no special authority.
By Manish Trivedi4 min read
Explanation is another generation task
Ask a person why they made a decision and you get an account that is partly memory and partly reconstruction, which psychology has documented at length. Ask a language model the same thing and you get something with no memory component at all. The system has no log of its own computation to consult.
What it does instead is produce text that plausibly follows from the question and the preceding answer. That is the only operation available to it. The explanation is generated in exactly the way the answer was generated, from the same statistics, with the same absence of any verification step.
This means an explanation can be entirely coherent, internally consistent, and unrelated to whatever process actually produced the output. It is not lying, since that would require an intent it does not have. It is doing the only thing it does.
Stated confidence is not measured confidence
There is a real internal quantity here — the probability the model assigned to each option — and it does carry information about reliability, imperfectly. But the number a model states when asked how sure it is has no direct connection to that quantity. It is produced as text, conditioned on the conversation, and shaped by whatever expressions of confidence appeared in training data and were rewarded afterwards.
The result is that verbal confidence tends to be poorly calibrated, and in a specific direction: it is often high when the topic is unusual, because unusual topics do not come with linguistic markers of uncertainty attached. Confidence stated in a percentage looks precise and is one of the least trustworthy outputs available.
Research on eliciting better-calibrated uncertainty is active and has produced real improvements. It has not produced anything that should be treated as a reliable gauge, and the underlying obstacle — that the report and the state being reported are only loosely coupled — has not gone away.
Reasoning traces are output, not process
Systems that write out their working before answering perform better on many problems, and this is a robust and well-replicated finding. The improvement is real and there is a mechanical explanation for it: the intermediate text becomes part of the input for subsequent steps, so the computation available to the final answer is genuinely greater.
It does not follow that the written trace describes the computation. Work examining this has found cases where a model’s stated reasoning omits a factor that demonstrably influenced its answer, or presents a justification constructed after the conclusion was effectively fixed. The trace is text that helps produce the answer, which is not the same as text that reports how the answer was produced.
This distinction matters most where the trace is being used for oversight. Reading the reasoning to check whether a system is behaving acceptably assumes the reasoning is faithful, and faithfulness is a property that has to be tested rather than assumed. It is currently an active research question with no settled answer.
Agreement under pressure is a related symptom
A well-known behaviour is that these systems frequently revise a correct answer when a user pushes back, even when the pushback contains no argument. The mechanism is straightforward: disagreement in the conversation shifts the statistics towards concession, and training on human preferences rewards being agreeable.
The consequence for self-reports is direct. If a stated explanation can be talked out of existence by mild scepticism, the explanation was not anchored to anything. Testing this is easy and worth doing, since a system that abandons its reasoning under pressure was never reporting reasoning in the first place.
Providers work on this deliberately and behaviour differs considerably between systems and versions. It is a tendency rather than a law, and it can be reduced. It is also a good reminder of what the underlying object is.
Why the field is spending real effort here
Interpretability research attempts to get at the actual computation rather than the model’s account of it — examining internal activations, identifying structures that correspond to recognisable concepts, and intervening on them to see whether behaviour changes as predicted. Some of this work has produced results that are genuinely surprising and genuinely useful.
It is also difficult, partial and contested. There is disagreement about whether current techniques recover real structure or impose it, about whether findings on small models transfer to large ones, and about how far this line of work can go in principle. Nobody involved claims the problem is close to solved.
The practical takeaway is narrow and firm. Whatever a system says about itself is an output like any other. Treat it as a hypothesis worth testing, never as testimony, and be especially careful with it in exactly the situations where an explanation would be most reassuring.
Common questions
So is asking a model to explain itself useless?
Not useless, but the value is different from what it appears to be. A generated explanation can surface a consideration you had not thought of, and it can be checked independently. What it cannot do is tell you why the system produced its answer, and reading it as though it could is the error.
Can you get a real confidence number out of a model?
With direct access to the model you can read the probabilities it assigned, and those are better calibrated than anything it says out loud, though still imperfect after later training stages. Through an ordinary conversational interface that information is generally unavailable.
Why do models sometimes admit a mistake that was not a mistake?
Because the conversation now contains a challenge, and text following a challenge is statistically more likely to concede than to hold firm. The system is continuing a plausible exchange rather than re-examining the question, which is why an unargued objection works about as well as an argued one.
Consumer editor, AI Worth Knowing
Manish has written about how it works, in the world, limits & risks for most of the last decade and prefers a plain explanation to a clever one.
