Where it comes from and what it does.
WHERE BIAS ORIGINATES
The training data, reflecting historical patterns including unfair ones The labels, reflecting the judgements of whoever assigned them The choice of what to measure Unrepresentative data, so some groups are poorly served
WHAT IT LOOKS LIKE IN PRACTICE
A recruitment system preferring candidates resembling past hires A facial recognition system performing worse on some groups A credit model reproducing historical lending patterns A language model reflecting stereotypes present in its training text
WHY IT IS NOT SIMPLY FIXED
Removing a sensitive attribute does not remove the bias. Other variables correlate with it.
A model with no race field can still produce racially disparate outcomes through postcode.
WHAT HELPS
Representative training data Testing outcomes across groups, not just overall accuracy Human review of consequential decisions Being able to explain decisions
THE RESPONSIBILITY
Sits with whoever deploys the system, not with the system.
WHAT TO ASK
How was this tested across different groups of people?
