Understanding where error comes from.
WHAT BIAS IS
Error from a model too simple to represent the pattern.
WHAT VARIANCE IS
Error from sensitivity to the particular training sample.
WHAT IRREDUCIBLE ERROR IS
Noise that no model can predict.
WHY THAT MATTERS
It sets a ceiling, and pursuing accuracy beyond it is wasted effort.
HOW TO ESTIMATE IT
Human performance on the same task, where applicable.
WHAT ERROR ANALYSIS IS
Examining individual wrong predictions to find patterns.
WHY IT IS THE MOST VALUABLE ACTIVITY
It shows what to fix, which metrics alone never do.
HOW TO DO IT
Take a sample of errors, categorise them, and count.
WHAT THAT REVEALS
Which category of error is most common, and therefore most worth addressing.
WHAT CATEGORIES COMMONLY EMERGE
Mislabelled training data A class the model rarely saw Input types nobody anticipated Genuinely ambiguous cases
WHAT TO DO ABOUT MISLABELLED DATA
Fix it, since it limits achievable performance.
WHAT TO DO ABOUT AMBIGUOUS CASES
Accept them, and stop optimising against them.
WHAT TO NEVER DO
Chase a metric without examining what is failing.
