Estimates you can trust.
WHAT UNDERMINES AN ESTIMATE
Tuning against the test set Selecting the best of many attempts, then reporting it Leakage An unrepresentative test set Evaluating on resampled data
WHAT SELECTING THE BEST OF MANY CAUSES
An optimistic estimate, since some of the improvement is chance.
WHAT PROTECTS AGAINST IT
A final test set used once.
WHAT TO ESTIMATE
Uncertainty in the metric, not only the point value.
HOW
Resampling, or repeated splits.
WHY IT MATTERS
A difference smaller than the uncertainty is not a difference.
WHAT TO COMPARE AGAINST ALWAYS
The baseline, and the current system if one exists.
WHAT TO EVALUATE ON
The distribution the model will actually face.
WHAT THAT MEANS FOR TIME
Test on the most recent period, not a random sample.
WHAT TO ALSO MEASURE
Performance on subgroups Behaviour on unusual inputs What happens when a feature is missing
WHAT TO REPORT
The metric, its uncertainty, the baseline, and the limitations.
WHAT TO NEVER REPORT
A single number with no context.
