Unfair patterns in output and decisions.
THE DEFINITION
Systematic unfairness in a system's output or decisions, reflecting patterns in its training data.
WHERE IT COMES FROM
Training data reflecting historical patterns, including unfair ones Labels reflecting the judgements of whoever assigned them Unrepresentative data, so some groups are served poorly
WHY REMOVING A FIELD DOES NOT FIX IT
Other variables correlate. Postcode, employer, school and name all carry information about protected characteristics.
A system with no race field can produce racially disparate outcomes.
WHAT THIS MEANS PRACTICALLY
Testing overall accuracy tells you nothing about fairness.
Test outcomes by group.
WHOSE RESPONSIBILITY
Whoever deploys the system, not the system.
WHAT TO ASK ANY VENDOR
How was this tested across different groups of people?
RELATED TERMS
Fairness, training data, explainability.
