The error that ruins everything.
WHAT LEAKAGE IS
Information in training data that would not be available at prediction time.
WHY IT IS THE WORST ERROR
The model appears excellent and fails completely in production.
WHAT CAUSES IT
Features derived from the outcome Features computed after the prediction point Aggregates computed over the whole dataset before splitting Identifiers correlating with the target Duplicate records across splits
WHAT AN OBVIOUS EXAMPLE LOOKS LIKE
A field populated only after the outcome occurred.
WHAT A SUBTLE EXAMPLE LOOKS LIKE
A record's update timestamp, which changes when the outcome is recorded.
WHAT SIGNALS IT
Performance far better than seems plausible One feature dominating importance Performance collapsing in production
WHAT TO DO ON SUSPICION
Examine the top features, and ask when each becomes known.
WHAT TO ESTABLISH FOR EVERY FEATURE
The moment its value becomes available.
WHAT TO DO ABOUT PREPROCESSING
Fit it on training data only, then apply to validation and test.
WHY
Scaling using the whole dataset's statistics leaks.
WHAT TO CHECK FOR
Duplicates spanning splits.
