Calling language and vision models.
WHAT MAKES THEM DIFFERENT
Responses vary between identical requests Latency is high and variable Cost is per unit of text processed Output may be wrong while appearing confident
WHAT TO SET
Generous timeouts Streaming, where responsiveness matters Limits on output length
WHAT TO BUDGET FOR
Cost per request, multiplied by expected volume.
WHY THAT CALCULATION MATTERS
It frequently determines whether the feature is viable.
WHAT TO CACHE
Responses to identical inputs, where appropriate.
WHAT TO NEVER SEND
Data you are not permitted to share with a third party.
WHY
It leaves your control, and obligations may apply.
WHAT TO CHECK
The provider's terms on data retention and training.
WHAT TO VALIDATE
Every response, before using it.
WHY
Structure is not guaranteed, even when requested.
WHAT TO DO ABOUT STRUCTURED OUTPUT
Request a specific format, and parse defensively.
WHAT TO HANDLE
Responses that do not parse Content that is wrong Refusals
WHAT TO BUILD
A fallback for when the model is unavailable.
WHAT TO MONITOR
Cost Latency Error and refusal rates
WHAT TO BE CAREFUL WITH
Passing user input directly into instructions.
WHY
It permits the input to redirect the model's behaviour.
