Data readiness.
WHAT TO ESTABLISH
Whether examples of the outcome exist, historically How many Whether the features were available at prediction time Whether the data is labelled, or can be How representative it is of future conditions
WHAT VOLUME IS REQUIRED
It depends on the task, but the number of examples of the rarer class matters more than the total.
WHY
A million rows with fifty positive examples is fifty examples.
WHAT CLASS IMBALANCE MEANS
One outcome vastly outnumbering another.
WHERE IT IS COMMON
Fraud, failure prediction, rare disease, conversion.
WHAT IT BREAKS
Accuracy as a metric, and training that ignores the rare class.
WHAT TO CHECK ABOUT REPRESENTATIVENESS
Whether the process generating the data has changed Whether the population has changed Whether historical decisions influenced what was recorded
WHAT THAT LAST POINT MEANS
If loans were only granted to people who repaid, you cannot learn who would have repaid.
WHAT THAT IS CALLED
Selection bias, and it is extremely common.
WHAT TO DO ABOUT IT
Acknowledge it, and be careful what you conclude.
WHAT TO DO IF DATA IS INADEQUATE
Say so, and propose collecting it.
