A recommender has to guess the question, and that decides everything about its design

One system is answered a question, the other has to invent it

A search engine receives a request. However clumsy the words, the person has said something about what they want, and the system’s job is to find material matching it. Ambiguity remains, but the target is at least nominated. That nomination is worth more than any amount of clever machinery downstream.

A recommender receives nothing of the kind. Somebody opened an application, and the system must decide what to put in front of them without being told. Everything it uses instead — what this person did before, what similar people did, what is popular now, what time it is — is a substitute for the request that was never made.

This is why the two are built differently despite both producing ranked lists. Search is a matching problem with a query at its centre. Recommendation is an estimation problem, guessing at a preference that exists only inside somebody’s head and has to be inferred from traces.

The classic formulation is a table with most of it missing

Imagine a grid with people down one side and items across the top, and in each cell some measure of how much that person liked that item. Almost every cell is empty, because almost nobody has encountered almost anything. Recommendation, stated formally, is the problem of filling in the empty cells plausibly.

The insight that made this tractable is that the grid has far less structure in it than its size suggests. Preferences cluster. If a small number of underlying factors explain most of the pattern — genre, tone, difficulty, whatever the factors turn out to be, since they are discovered rather than named — then each person and each item can be described by a short list of numbers, and any cell estimated by combining the two.

This is where the idea of learning positions in a shared space entered mainstream practice, and it arrived through recommendation before it arrived anywhere more celebrated. The factors themselves are not interpretable in any reliable way, which has never stopped people from telling stories about what they mean.

Two families, and the gap between them is the cold start

The approach above uses only behaviour: who interacted with what. It needs no understanding of the items at all, which is its great strength — it works equally well for films, groceries and job listings — and it fails completely for anything new. A film nobody has watched has no column worth reading.

The alternative describes items by their content and recommends things resembling what a person liked before. It handles new items without difficulty, and it tends to produce narrower results, because similarity of content is a poorer predictor of enjoyment than the tastes of people who resemble you. Most real systems combine both, weighted by how much behavioural evidence exists.

The genuine difficulty is a new person rather than a new item. With no history, the system falls back on whatever is generally popular, which is why the first experience of any such service is a poor guide to what it will eventually do.

At scale the work is split into two very different stages

No system scores every item for every request, because catalogues are far too large for that. Instead a fast, crude stage retrieves a manageable shortlist from millions of candidates using cheap approximations, and a slower, more elaborate stage ranks that shortlist properly. The two stages have different objectives and are usually built by different people.

This split has a consequence that is easy to miss. Anything the first stage does not retrieve cannot be recommended, however good the second stage is. A great deal of what a system will never show you is decided by an approximation chosen for speed, not by any judgement about quality.

The final ordering is then usually adjusted again — for freshness, for variety, for business rules, for whatever the operator has decided must appear. By that point the trained model is one input among several, which complicates every attempt to explain a particular result by reference to the model alone.

The objective is where the real argument lives

A recommender optimises something specific and measurable: a click, a purchase, minutes spent, a rating. None of those is the same as the thing anybody would say they wanted, and the substitution is not a technical detail. It is the whole design, expressed as a number that engineering can act on.

Choosing differently is possible and it happens. Systems are built to value variety, to discount things a person would have found anyway, to weight a considered signal above an impulsive one. These choices are real and they trade measurable engagement for something harder to demonstrate, which is why they are argued about internally rather than settled.

What the mechanism makes clear is that there is no neutral default. A system that has to guess the question is always answering some question, and which one it answers was decided by whoever chose what to measure.