A model trained on past decisions will reproduce the reasoning behind them

What these systems are actually asked to do

A screening model is trained on historical cases with known outcomes and asked to predict the outcome for a new case. Applications that led to a hire, loans that were repaid, claims that turned out valid. The model finds the statistical patterns that separated one group from the other and applies them to whoever arrives next.

Framed that way, the appeal is obvious. It is consistent, it is fast, it does not get tired at four in the afternoon, and it can be applied to volumes that no panel of people could read. Those are genuine advantages and the case for automation is not frivolous.

The difficulty is buried in the phrase historical cases with known outcomes. Every part of that phrase is doing more work than it appears to.

The label is rarely the thing you wanted to measure

Suppose you want a model that identifies people who would perform well in a role. You do not have that data. What you have is data about people who were hired, and how their performance was rated by managers, and whether they stayed. Everyone who was rejected has no outcome at all, and their absence is not random.

So the model learns to predict who resembles people who were previously hired and subsequently rated well. If the earlier selection favoured a particular background, or if the performance ratings themselves reflected who managers found easy to work with, the model reproduces that faithfully. It is doing its job correctly. The job was mis-specified.

This substitution problem appears in every domain where these systems are used. Repayment records reflect who was offered credit on what terms. Enforcement records reflect where enforcement happened. The measured quantity is downstream of past choices, and the model has no way of knowing that.

Removing the sensitive field does not remove the information

The intuitive fix is to withhold the attributes that should not matter — do not give the model a gender field, an ethnicity field or an age. This is necessary and it is nowhere near sufficient, because in a rich dataset those attributes are reconstructable from other variables.

Postcode carries information about ethnicity and income in most countries. Institution attended carries information about class and often geography. Employment gaps correlate with caregiving. A model does not need the protected field; it needs only variables that happen to be correlated with it, and in a large feature set there will be many.

Worse, removing the field can make the problem harder to detect. You cannot measure whether outcomes differ across a group if you have deliberately deleted the ability to identify the group, which is why several regulatory approaches now require collecting sensitive attributes for auditing while forbidding their use in the decision.

The fairness definitions genuinely conflict

There is more than one reasonable definition of a fair decision procedure. One says the error rates should be equal across groups. Another says that among people given the same score, the actual rate of the outcome should be the same across groups. A third says similar individuals should receive similar treatment.

It has been shown that under ordinary conditions — where the underlying rate of the outcome genuinely differs between groups, for whatever historical reason — these criteria cannot all be satisfied at once. This is a mathematical result, not a political opinion, and it means that choosing a fairness standard is choosing which unfairness to accept.

That conclusion is unwelcome to almost everyone. It is also the single most useful thing to know when reading a claim that some system has been made fair, because the honest version of that claim always specifies which definition was used.

The comparison that matters is not with perfection

A common defence of automated screening is that the human process it replaced was inconsistent, opaque and demonstrably biased in its own right, and that at least a model can be audited. There is real force in this. Human decisions vary with time of day, mood and order of presentation, and nobody can inspect the weights inside a hiring manager.

The counter-argument is about scale and appeal. One biased assessor affects the applications they see; one biased model affects every application, identically, with no variation that might let an unusual candidate through. And a person can be asked to explain a decision, whereas a model’s explanation, if one is produced at all, is generally a reconstruction rather than a record.

Which consideration dominates depends on the setting, the volume and what recourse exists. The framing that helps is to stop asking whether the system is biased, which it will be in some respect, and start asking who can find out, and what happens when they do.