Fairness and bias
Not to be confused with Bias-variance trade-off — that “bias” is underfitting. This one is about a model treating groups differently, and it is a legal obligation in credit, insurance, employment and housing before it is an ethical position.
Removing the attribute does not remove the bias
The first instinct is to drop race or gender from the features. It does
not work, because other features encode them:
postcode ─┐
first name ├─▶ proxies for the
device type │ protected attribute
shopping data ─┘A model with enough features will reconstruct the attribute you deleted. This is why “we don’t collect it, so we can’t be biased” is not a defence — and why you often need to collect the attribute in order to measure fairness, which is the uncomfortable part of the conversation.
The metrics disagree, and that is a theorem
| Metric | Equal across groups |
|---|---|
| Demographic parity | positive rate |
| Equal opportunity | true positive rate |
| Equalised odds | TPR and FPR |
| Predictive parity | precision |
You cannot satisfy all of them at once unless base rates are equal or the classifier is perfect. That is an impossibility result, not an engineering gap, and knowing it is the difference between a considered answer and a naive one.
So the real question is which definition your domain requires:
- Lending typically cares about equal opportunity — a creditworthy applicant should be approved at the same rate regardless of group.
- Screening for a rare disease cares about false negatives, so equalised odds matters and demographic parity would be actively wrong.
- Advertising reach is where demographic parity is often the legal standard, because the harm is exclusion from an opportunity.
Pick one, state why, and be explicit that the others move as a consequence.
Where the bias comes from
Fixing the model is usually the wrong layer.
- Historic labels. If past decisions were biased, a model trained to reproduce them is working correctly and is still wrong. Hiring data trained on who was previously hired is the canonical case.
- Sampling. Underrepresented groups get worse accuracy simply from having less data — the model is optimising average loss, and they are a small part of the average.
- Label definition. “Good customer” defined as high spend encodes wealth. The proxy you chose is the bias.
- Feedback loops. A model that scores a group lower produces less data about them, which makes the next model worse. Policing and lending both have documented versions.
Point three is the one interviews reward, because it is upstream of everything technical.
Measuring it
from fairlearn.metrics import (
MetricFrame, selection_rate,
)
from sklearn.metrics import recall_score
mf = MetricFrame(
metrics={
"recall": recall_score,
"sel": selection_rate,
},
y_true=y_test, y_pred=y_pred,
sensitive_features=group,
)
mf.by_group # per-group metrics
mf.difference() # the gap that mattersThe habit worth having: slice every headline metric by group as a matter of course, the same way you would slice by segment for performance. Aggregate accuracy hides a model that works for 90% of users and fails the other 10%.
Gotcha: with a small group, the metric gap has a wide confidence interval. A 4-point difference on 60 people is noise. Report intervals, or you will chase a phantom and “fix” it into a real one.
Mitigation, by where it acts
| Stage | Approach |
|---|---|
| Pre-processing | reweight or resample the training data |
| In-processing | a fairness constraint in the objective |
| Post-processing | per-group thresholds |
Post-processing is the most effective and the most legally fraught: setting a different decision threshold per group is explicitly disparate treatment in some jurisdictions even when it produces a fairer outcome. That tension — fairness through awareness versus the law’s suspicion of group-based rules — is worth naming rather than resolving.
The safest lever is usually upstream: fix the label, fix the sampling, and remove the proxy features you cannot justify.
What the regulation asks for
Under the EU AI Act, credit scoring and employment screening are high-risk, which brings obligations on data governance, documentation, human oversight and post-market monitoring. High-risk obligations phase in through 2027-2028.
The practical consequence for an engineer: keep the evaluation artefacts, the slice metrics and the decision log. The audit asks for evidence rather than intentions, and evidence is something you either recorded at the time or did not. See PII, privacy and the EU AI Act.
Related
Interview angle 5
- “Can you make a model fair by dropping the protected attribute?” - no. Postcode, name, device and purchase history are proxies, and a model with enough features reconstructs what you deleted. You often have to collect the attribute in order to measure fairness at all.
- “Which fairness metric would you use?” - whichever the domain requires, stated explicitly. Equal opportunity for lending, equalised odds where false negatives cause harm, demographic parity where exclusion is the harm. You cannot satisfy them simultaneously unless base rates are equal — that is an impossibility result, not an implementation gap.
- “Where does the bias actually come from?” - usually the labels, not the model. A model trained to reproduce biased past decisions is working correctly and still wrong. After that: sampling, the proxy you chose for the target, and feedback loops.
- “How do you detect it?” - slice every headline metric by group as routine practice, with confidence intervals. Aggregate accuracy hides a model that fails a tenth of users, and a gap measured on 60 people is usually noise.
- “How would you fix it?” - prefer upstream: the label definition, the sampling, and unjustifiable proxy features. Per-group thresholds are the most effective post-hoc fix and the most legally fraught, since a group-based rule can itself be disparate treatment.