What the system learns from.
THE PRINCIPLE
A model is shaped entirely by what it was trained on.
Its capabilities, its blind spots and its biases all come from there.
WHAT GOOD TRAINING DATA LOOKS LIKE
Representative of the situations the system will face Accurate labels, where labels are used Sufficient quantity Diverse enough to cover edge cases
WHAT GOES WRONG
Data unrepresentative of real use, so the system fails in production Historical bias reproduced and amplified Insufficient examples of rare but important cases Errors in labels, which the system learns as truth
THE CLASSIC FAILURE
A hiring model trained on past hiring decisions learns past preferences, including discriminatory ones.
The model is not biased in itself. It faithfully reproduced what it was shown.
FOR LANGUAGE MODELS
Trained on very large amounts of text, mostly from the internet.
That determines what they know, how they write, and what they get wrong.
THE PRACTICAL IMPLICATION
Ask what a system was trained on when the answer matters.
