The summary.
MONITOR INPUT DISTRIBUTIONS, BECAUSE OUTCOMES ARRIVE LATE OR NEVER
Actual performance may not be measurable for weeks, so distribution shift is the early signal available today.
Record the training distribution at training time — you cannot compare against what you did not save.
CONCEPT DRIFT IS WORSE THAN DATA DRIFT
The learned relationship becomes wrong, and input monitoring does not reveal it directly. Investigate the cause before retraining, or you entrench the problem.
Acting on predictions changes behaviour, which shapes the data the model then sees.
NEVER DEPLOY A RETRAINED MODEL WITHOUT EVALUATING IT
Retraining can produce a worse model, especially on drifted or broken data. Gate automated promotion on comparison with the current model.
THREE THINGS CHANGE IN ML DELIVERY, NOT ONE
Code, data and model. Version all three, and ensure every production model can be reproduced from what is recorded.
BUILD NOTHING AS A PLATFORM UNTIL ONE MODEL HAS SHIPPED MANUALLY
Platforms built before use encode assumptions that turn out wrong.
INFERENCE USUALLY DOMINATES COST, BECAUSE IT RUNS CONTINUOUSLY
Compare each model's cost against the value its predictions produce, and retire the ones that fail that test.
