Finding good settings.
WHAT A HYPERPARAMETER IS
A setting chosen before training, not learned from data.
WHAT THE COMMON ONES ARE
Learning rate Regularisation strength Model size and depth Batch size Tree depth and count, for ensembles
WHAT GRID SEARCH DOES
Tries every combination from defined lists.
WHY IT IS INEFFICIENT
Most parameters matter little, and effort is spent varying them.
WHAT RANDOM SEARCH DOES
Samples combinations randomly.
WHY IT OUTPERFORMS GRID SEARCH
It explores more values of the parameters that matter.
WHAT BAYESIAN APPROACHES DO
Model the relationship between settings and performance, choosing promising points.
WHEN THAT IS WORTH IT
Expensive training runs.
WHAT EARLY STOPPING OF TRIALS PROVIDES
Abandoning unpromising runs, testing more settings in the same time.
WHAT TO TUNE FIRST
The learning rate, for neural networks Depth and learning rate, for boosting
WHAT TO TUNE ON
Validation data, never test.
WHAT TO RECORD
Every trial, its settings and its result.
WHY
So the search can be resumed and understood.
WHAT TO BE CAREFUL WITH
Tuning so extensively that validation performance becomes optimistic.
