Training on graphics processors.
WHAT DETERMINES WHAT YOU CAN TRAIN
Memory, primarily.
WHAT CONSUMES IT
Model parameters Gradients and optimiser state Activations retained for the backward pass The batch
WHY OPTIMISER STATE IS OFTEN OVERLOOKED
It can exceed the model's own size several times over.
WHAT REDUCES MEMORY USE
Smaller batches Gradient accumulation, simulating larger batches Mixed precision Checkpointing activations, trading computation for memory Parameter-efficient fine-tuning
WHAT GRADIENT ACCUMULATION DOES
Processes several small batches before updating, achieving the effect of a large batch.
WHAT TO CHECK WHEN UTILISATION IS LOW
The data pipeline, which is usually the cause.
WHY
The device waits while data is prepared.
WHAT FIXES IT
Loading in parallel Preprocessing in advance Storing data in a format read efficiently
WHAT TO MONITOR DURING TRAINING
Device utilisation and memory Temperature, on local hardware
WHAT TO CONSIDER LOCALLY
Renting capacity rather than purchasing Power stability, which affects long training runs
WHAT TO ALWAYS IMPLEMENT
Checkpointing, so an interruption costs minutes rather than days.
