Controlling spend.
WHAT DRIVES COST
Training runs Inference volume Hardware type Data storage and transfer Managed service charges
WHAT DOMINATES OVER TIME
Inference, usually, since it runs continuously.
WHAT REDUCES INFERENCE COST
A smaller model Quantisation Caching repeated inputs Batch rather than request-based, where possible Processors rather than accelerators, for small models
WHAT REDUCES TRAINING COST
Transfer learning Sampling during development Early stopping Interruptible capacity
WHAT TO MEASURE
Cost per thousand predictions Cost per training run Total by model
WHY PER MODEL
It reveals which are worth their cost.
WHAT TO COMPARE COST AGAINST
The value the predictions produce.
WHAT THAT REVEALS
Models costing more than the decisions they improve are worth.
WHAT TO DO ABOUT THOSE
Retire them.
WHAT TO SET
Budget alerts, from the start.
WHY THAT MATTERS HERE
Charges are in foreign currency and accumulate continuously.
WHAT TO REVIEW
Whether each deployed model is still used.
