Models on mobile and embedded devices.
WHAT IT PROVIDES
Running models on devices, without a server.
WHAT THAT PROVIDES
No connectivity requirement No data leaving the device No per-request cost Low latency
WHY THAT MATTERS PARTICULARLY HERE
Data is metered, connectivity is intermittent, and server costs are in foreign currency.
WHAT CONVERSION INVOLVES
Transforming a trained model into a compact format.
WHAT QUANTISATION DOES
Reduces numerical precision, making the model smaller and faster.
WHAT IT COSTS
Some accuracy.
WHAT TO MEASURE AFTER CONVERTING
Accuracy, against the original Model size Inference time, on a modest device
WHY A MODEST DEVICE
Performance varies enormously with hardware, and the slowest device determines the experience.
WHAT TO CONSIDER ABOUT APPLICATION SIZE
Bundled models add substantially to the download.
WHAT TO DO ABOUT IT
Download the model after installation, with a fallback.
WHAT TO HANDLE
Download failure Inference slower than expected Hardware acceleration being unavailable
WHAT TO TEST
On the oldest device you support.
