Use these patterns when moving from prototype to production.
Reliability checklist
- Set client-side timeout and cancellation controls.
- Handle
429 and 5xx with exponential backoff.
- Treat disconnected streams as retryable.
- Persist partial text only after completion if correctness matters.
Browser/Node cancellation example
UI guidance
- Render token deltas progressively.
- Show an explicit “generation ended” state when
[DONE] arrives.
- Keep request id and model id in logs for replay/debugging.
When not to stream
- Tool-calling flows under current request validation (
stream + tools rejected).
- Workflows requiring full validation before any user-visible output.
Last modified on July 26, 2026