When one outcome is rare.
WHY IT MATTERS
A model predicting the majority class always can appear highly accurate while being useless.
WHAT TO NEVER USE AS THE METRIC
Accuracy.
WHAT TO USE INSTEAD
Precision and recall The area under the precision-recall curve Metrics weighted by the cost of each error type
WHAT RESAMPLING DOES
Changes the training distribution: removing majority examples or duplicating minority ones.
WHAT SYNTHETIC GENERATION DOES
Creates new minority examples by interpolation.
WHAT TO BE CAREFUL WITH
Resampling before splitting, which leaks information Synthetic examples that are unrealistic
WHAT CLASS WEIGHTING DOES
Makes errors on the rare class cost more during training.
WHY THAT IS FREQUENTLY PREFERABLE
It changes nothing about the data, and is simpler to reason about.
WHAT THRESHOLD ADJUSTMENT DOES
Changes the point at which a probability becomes a positive prediction.
WHY THAT IS THE MOST UNDERRATED TOOL
The default threshold is arbitrary, and adjusting it frequently achieves more than any resampling.
WHAT TO SET IT FROM
The relative cost of the two error types.
WHAT TO EVALUATE ON
The original distribution, never the resampled one.
