The summary.
DEFINE THE TARGET PRECISELY BEFORE ANYTHING ELSE
Predicting churn requires defining churn, and the definition changes the problem entirely. Agree the metric and the usefulness threshold before building, not after.
THE NUMBER OF RARE-CLASS EXAMPLES MATTERS, NOT THE TOTAL
A million rows with fifty positive examples is fifty examples.
Never use accuracy on imbalanced data — and threshold adjustment is the most underrated tool available, since the default threshold is arbitrary.
LEAKAGE MAKES A MODEL LOOK EXCELLENT AND FAIL COMPLETELY
Establish when each feature's value becomes available. Performance far better than seems plausible is the signal.
Fit preprocessing on training data only — scaling with whole-dataset statistics leaks.
SPLIT BY TIME FOR TIME-ORDERED DATA AND BY ENTITY WHERE ROWS SHARE ONE
Random splitting otherwise lets the model learn from the future, or see the same entity on both sides.
A MODEL CANNOT EXCEED THE CONSISTENCY OF ITS LABELS
Check inter-annotator agreement on a small set before labelling a large one.
IF HISTORICAL DECISIONS DETERMINED WHAT WAS RECORDED, YOU CANNOT LEARN THE COUNTERFACTUAL
