Practical training.
WHAT THE LOOP IS
Forward pass, compute loss, compute gradients, update parameters, repeat.
WHAT AN EPOCH IS
One pass through the training data.
WHAT BATCH SIZE AFFECTS
Memory use Training speed Gradient noise, which affects generalisation
WHAT LARGE BATCHES REQUIRE
Higher learning rates, and usually a warm-up period.
WHAT THE LEARNING RATE CONTROLS
How much parameters change per update.
WHY IT IS THE MOST IMPORTANT SETTING
Too high diverges; too low never converges.
WHAT A SCHEDULE DOES
Changes it during training, typically decreasing.
WHAT WARM-UP PROVIDES
Stability at the start, when gradients are large.
WHAT TO MONITOR DURING TRAINING
Training and validation loss, together.
WHAT DIVERGENCE BETWEEN THEM SHOWS
Overfitting beginning.
WHAT TO DO
Stop, and keep the best checkpoint.
WHAT MIXED PRECISION PROVIDES
Faster training and less memory.
WHAT IT REQUIRES
Loss scaling, to prevent small gradients vanishing.
WHAT TO CHECK BEFORE A LONG RUN
That the model can overfit a tiny sample.
WHY
If it cannot, something is broken.
