Controlling how much a model can learn.
WHAT CAPACITY IS
How complex a pattern a model can represent.
WHAT TOO LITTLE CAUSES
Underfitting: the model cannot capture the pattern.
WHAT TOO MUCH CAUSES
Overfitting: the model memorises rather than generalises.
WHAT INDICATES OVERFITTING
Training performance far exceeding validation performance.
WHAT INDICATES UNDERFITTING
Both being poor.
WHAT REGULARISATION DOES
Constrains the model, discouraging complexity.
WHAT THE COMMON METHODS ARE
Penalising large parameter values Randomly disabling parts during training Stopping training when validation performance stops improving Adding noise or augmenting data Reducing model size
WHAT PENALTY TYPES DIFFER IN
One shrinks parameters toward zero Another drives some to exactly zero, removing features
WHAT THAT SECOND EFFECT PROVIDES
Feature selection, as a side effect.
WHAT EARLY STOPPING REQUIRES
A validation set monitored during training.
WHAT MORE DATA DOES
Reduces overfitting more reliably than any regularisation technique.
WHAT THAT IMPLIES
Collecting more data is frequently the best available improvement.
