Why models decay.
WHAT DATA DRIFT IS
The input distribution changing.
WHAT CONCEPT DRIFT IS
The relationship between inputs and outcome changing.
WHY THE SECOND IS WORSE
The model's learned relationship becomes wrong, and no amount of input monitoring reveals it directly.
WHAT CAUSES DRIFT
Changes in customer behaviour Changes in the product Competitors and market changes Seasonal variation Upstream data changes The model's own effect on behaviour
WHAT THAT LAST POINT MEANS
Acting on predictions changes what happens, so the data the model sees is shaped by the model.
WHAT THAT IS CALLED
A feedback loop, and it invalidates naive evaluation.
WHAT SUDDEN DRIFT LOOKS LIKE
A step change, usually from a system or process change.
WHAT GRADUAL DRIFT LOOKS LIKE
Slow degradation, easily missed.
HOW TO DETECT DATA DRIFT
Statistical comparison of distributions against training.
HOW TO DETECT CONCEPT DRIFT
Performance measurement, once outcomes are known.
WHAT TO DO ON DETECTION
Investigate the cause before retraining.
WHY
Retraining on broken data entrenches the problem.
