Running a trained model.
THE DEFINITION
Using a trained model to produce output, as distinct from training it.
WHAT HAPPENS DURING INFERENCE
The model takes your input and generates a response based on patterns learned during training.
WHAT DOES NOT HAPPEN
Learning. The model is unchanged by using it.
WHY THE DISTINCTION MATTERS
Training is enormously expensive and happens once.
Inference happens every time anyone uses the model, and it is what you pay for.
WHY RESPONSES TAKE TIME
Each word is generated in turn, with everything before it in view.
That is why output streams rather than appearing at once.
WHAT AFFECTS SPEED
Model size, output length, and how busy the service is.
WHERE YOU ENCOUNTER THE TERM
API documentation and pricing.
RELATED TERMS
Training, token, API.
