Separating training from assessment.
WHAT THE SPLITS ARE
- Training: what the model learns from
- Validation: used for tuning decisions
- Test: used once, for a final estimate
WHY THREE
Tuning against the test set contaminates it, and its estimate becomes optimistic.
WHAT TO NEVER DO
Make decisions based on test performance, then report that performance.
WHAT RANDOM SPLITTING ASSUMES
Examples are independent and identically distributed.
WHEN THAT IS FALSE
Time-ordered data Several rows per entity Grouped or related records
WHAT TO SPLIT BY FOR TIME-ORDERED DATA
Time, with training before validation before test.
WHY
Random splitting lets the model learn from the future.
WHAT TO SPLIT BY WHEN ROWS SHARE AN ENTITY
The entity, so the same one does not appear on both sides.
WHAT CROSS-VALIDATION PROVIDES
Several splits, averaged, giving a more stable estimate.
WHEN IT IS WORTH IT
Small datasets, where a single split is noisy.
WHEN IT IS NOT
Large datasets, or expensive training.
WHAT TO KEEP FIXED
The split, so results are comparable across experiments.
