Making prediction fast and affordable.
WHAT TO MEASURE
Latency, at the percentiles that matter Throughput Cost per thousand predictions Memory use
WHY PERCENTILES
Average latency conceals the slow requests users notice.
WHAT QUANTISATION DOES
Reduces numerical precision of parameters.
WHAT IT PROVIDES
Smaller models, faster inference, less memory.
WHAT IT COSTS
A small accuracy reduction, usually.
WHAT PRUNING DOES
Removes parameters contributing little.
WHAT DISTILLATION DOES
Trains a smaller model to reproduce a larger one's behaviour.
WHY THAT IS POWERFUL
It frequently retains most of the performance at a fraction of the cost.
WHAT CACHING PROVIDES
Reusing predictions for repeated identical inputs.
WHERE THAT APPLIES
Inputs that recur, which is more common than expected.
WHAT HARDWARE CHOICE AFFECTS
Cost far more than latency, for small models.
WHAT TO ESTABLISH
Whether accelerated hardware is justified at your volume.
WHY
It frequently is not, and processors serve small models economically.
WHAT TO OPTIMISE FIRST
The preprocessing, which is often slower than the model.
WHAT TO PROFILE
The whole path, not the model alone.
