When predicted probabilities must be believable.
WHAT CALIBRATION MEANS
Among predictions of seventy per cent, about seventy per cent are positive.
WHEN IT MATTERS
Probabilities used in expected-value calculations Thresholds set from costs Outputs shown to people as likelihoods Predictions combined with other estimates
WHAT MODELS ARE POORLY CALIBRATED BY DEFAULT
Tree ensembles, which push probabilities toward extremes Support vector methods, which produce scores rather than probabilities
WHAT IS BETTER CALIBRATED
Logistic regression, generally.
HOW TO ASSESS IT
A calibration plot: predicted probability against observed frequency, in bins.
WHAT A WELL-CALIBRATED PLOT LOOKS LIKE
Close to the diagonal.
WHAT METHODS CORRECT IT
Fitting a simple model mapping scores to probabilities An isotonic mapping, non-parametric and more flexible
WHAT THEY REQUIRE
A held-out set, not used for training.
WHAT TO BE CAREFUL WITH
The flexible method overfitting on small data.
WHAT RECALIBRATION AFTER DEPLOYMENT MAY NEED
Redoing, as the population changes.
WHAT TO NEVER ASSUME
That a model's output is a probability simply because it lies between zero and one.
