Skip to main content
Use these patterns when moving from prototype to production.

Reliability checklist

  • Set client-side timeout and cancellation controls.
  • Handle 429 and 5xx with exponential backoff.
  • Treat disconnected streams as retryable.
  • Persist partial text only after completion if correctness matters.

Browser/Node cancellation example

UI guidance

  • Render token deltas progressively.
  • Show an explicit “generation ended” state when [DONE] arrives.
  • Keep request id and model id in logs for replay/debugging.

When not to stream

  • Tool-calling flows under current request validation (stream + tools rejected).
  • Workflows requiring full validation before any user-visible output.
Last modified on July 26, 2026