Starting from an existing model.
WHAT IT IS
Using a model trained on a large general dataset as the starting point.
WHY IT WORKS
Early layers learn general features useful across tasks.
WHAT IT PROVIDES
Useful performance from far less data Much shorter training Lower cost
WHAT THE APPROACHES ARE
Using the model as a fixed feature extractor Fine-tuning some layers Fine-tuning everything
WHAT TO CHOOSE
Feature extraction with very little data Partial fine-tuning with moderate data Full fine-tuning with substantial data and similar domain
WHAT LEARNING RATE TO USE WHEN FINE-TUNING
Lower than for training from scratch.
WHY
Large updates destroy what was learned.
WHAT CATASTROPHIC FORGETTING IS
Losing general capability while learning the new task.
WHAT REDUCES IT
Lower learning rates, fewer epochs, and freezing early layers.
WHAT PARAMETER-EFFICIENT METHODS PROVIDE
Adapting a large model by training a small number of additional parameters.
WHY THAT MATTERS
Full fine-tuning of large models is frequently unaffordable.
WHAT TO ALWAYS COMPARE AGAINST
The pre-trained model without any adaptation.
