The summary.
EVERY CLASSIFICATION METRIC DERIVES FROM THE CONFUSION MATRIX
Start there. Precision matters when acting wrongly is costly; recall matters when missing is costly.
The receiver curve looks good on imbalanced data even when precision is poor — use the precision-recall curve instead.
A MODEL'S OUTPUT IS NOT A PROBABILITY JUST BECAUSE IT LIES BETWEEN ZERO AND ONE
Tree ensembles push probabilities to extremes. Check the calibration plot before using outputs in any expected-value calculation.
REPORT UNCERTAINTY, NOT ONLY A POINT VALUE
A difference smaller than the uncertainty is not a difference. And selecting the best of many attempts produces an optimistic estimate.
THE LEARNING RATE IS THE MOST IMPORTANT SETTING
Too high diverges, too low never converges. Tune it first.
CHECK THAT THE MODEL CAN OVERFIT A TINY SAMPLE BEFORE ANY LONG RUN
If it cannot, something is broken — and finding that after hours of training is an avoidable expense.
CHECKPOINT THE OPTIMISER STATE, NOT ONLY THE PARAMETERS
Resuming without it degrades training.
