Where most GPU work happens now.
WHY GRAPHICS PROCESSORS SUIT IT
Training is dominated by large matrix operations, which are exactly what they are built for.
WHAT DETERMINES TRAINING SPEED
Arithmetic throughput Memory bandwidth Memory capacity Interconnect, when using several devices
WHAT MEMORY CAPACITY LIMITS
Model size and batch size.
WHAT HAPPENS WHEN IT IS EXCEEDED
Training fails, or must be restructured.
WHAT REDUCED PRECISION PROVIDES
Faster computation and less memory, with careful handling.
WHAT MIXED PRECISION MEANS
Computing in lower precision while keeping critical values in higher.
WHAT IT REQUIRES
Loss scaling, to prevent small values vanishing.
WHAT DISTRIBUTED TRAINING PROVIDES
Using several devices, or several machines.
WHAT THE APPROACHES ARE
Replicating the model and splitting data Splitting the model itself, where it does not fit
WHAT LIMITS SCALING
Communication between devices, which the interconnect determines.
WHAT TO MEASURE
Utilisation.
WHY
Low utilisation usually means the data pipeline cannot keep the device fed.
WHAT TO FIX FIRST IN THAT CASE
Data loading, not the model.
