Routing and Fallbacks

The Phaseo Gateway routes each request to a provider that can serve your chosen model. When a provider is slow, rate-limited, or returning errors, the Gateway can attempt fallbacks so your requests still complete.

Choose a routing mode

Start with balanced unless one production requirement clearly matters more than the others.

Mode	Use it when
`balanced`	You want a practical mix of price, latency, throughput, and availability.
`price`	Reducing provider cost matters more than minimizing response time.
`latency`	Fast response start is the primary requirement.
`throughput`	Sustained token generation speed matters most.

Configure the workspace default in Dashboard -> Settings -> Routing. Use presets when one workflow needs a narrower provider or model policy than the workspace default.

Model routing suffixes

Append a routing suffix when the model ID itself should make the optimization mode authoritative for that request:

Suffix	Routing mode
`:nitro`	`throughput`
`:cheap`	`price`
`:fast`	`latency`

For example, openai/gpt-5-mini:nitro uses throughput-first routing. A recognized routing suffix takes precedence over request-level routing.mode or provider.sort, preset and workspace routing modes. Other constraints such as provider allow lists, regional requirements, guardrails, and price ceilings still apply.

How routing works at a high level

You send a request with a model id.
The Gateway evaluates provider health, latency, and capability coverage.
A provider is selected and the request is executed.

Inspect request outcomes through your activity logs and response metadata when debugging routing behavior.

Choose an exact provider in the model id

Use <provider-id>:<canonical-model-id> when a request must use one specific provider-model pair:

{
  "model": "baseten:thinking-machines/inkling-small",
  "input": "Hello"
}

The qualifier disables cross-provider fallback for that request. Model suffixes remain part of the canonical model ID, including in identifiers such as baseten:google/gemma-4-26b-a4b:free. Read Provider-qualified model IDs for complete syntax, free-route validation, routing precedence, aliases, error codes, and request examples.

Control routing and fallbacks

The current public fallback and routing controls are intentionally explicit:

Presets constrain the fallback pool

In Dashboard -> Settings -> Presets, you can define:

allowed models
provider allow lists
provider ignore lists
default prompt and parameter behavior

Those constraints are applied before provider selection, so a preset can intentionally narrow which providers are eligible for retries and failover.

Routing mode changes provider ranking

In Dashboard -> Settings -> Routing, workspaces can tune how the Gateway ranks compatible providers:

balanced
price
latency
throughput

The same page also exposes beta and alpha channel toggles so preview traffic can be introduced intentionally instead of appearing as an untracked routing side effect.

BYOK fallback is explicit

In Dashboard -> Settings -> BYOK, teams can choose whether a failed BYOK request is allowed to fall back to Phaseo credits. This is the current public control for the common “my own key failed, should the request still complete?” decision.

Dynamic routes attach policy to API keys

In Dashboard -> Settings -> Routing, create a dynamic route when different API keys or request classes need different provider behavior. A route can:

branch on nested request-body fields, request headers, custom metadata, endpoint, model, or session ID
split traffic by percentage for A/B tests and gradual rollouts
enforce daily, weekly, or monthly request and cost limits using the authenticated key’s usage buckets
call a different model and choose its routing mode, provider preference, and fallback policy
enable cache-aware and session-aware provider affinity
attach to one or more inference API keys

Conditions expose true and false outputs. Rate and budget nodes expose within and exceeded outputs. Percentage selection is deterministic for a session or prompt-cache key, so a rollout does not randomly move the same cached conversation between branches. Saving creates an immutable draft version. Deploying a selected version copies that snapshot to the Gateway and invalidates policy caches for attached keys. Earlier versions remain available for rollback. Operational provider-health recommendations appear under Insights, separately from the flow editor. Custom metadata is available on the OpenAI-compatible text inference surfaces:

{
  "model": "openai/gpt-5-mini",
  "metadata": {
	    "customer_plan": "pro",
	    "workspace": "acme"
  }
}

Fallback behavior

If a provider returns errors or rate limits, the Gateway can retry or route to another provider that supports the same model. You should still handle 429 and 5xx responses with exponential backoff. Model nodes in a dynamic route can also define an ordered list of fallback models. Phaseo first exhausts the eligible provider attempts for the selected model. If the resulting response is retryable (429, 500, 502, 503, or 504), the Gateway reruns the complete policy and provider-selection pipeline for each fallback model in order. Client errors are returned immediately and do not switch models. Each fallback is independently checked against workspace model restrictions, guardrails, provider policy, pricing, and capability support. A route can store up to eight fallback models. Read more in:

Cache-aware and session-aware affinity

Cache-aware routing is enabled by default on text generation endpoints. After a provider reports a real prompt-cache read, Phaseo pins matching context back to that provider for 15 minutes, provided the provider remains healthy and allowed by the active route, preset, guardrail, and request policy. When session_id is present, the cache affinity is keyed to the session instead of only the opening context. Session affinity is refreshed when Phaseo observes another cache read and is retained for up to 24 hours of session activity. Circuit breakers and policy filters always take precedence over affinity. Opt out for one request without changing workspace or dynamic-route defaults:

{
  "model": "openai/gpt-5",
  "session_id": "support-session-42",
  "provider": {
    "cache_aware_routing": false
  }
}

To retain context cache affinity but ignore the session identifier for one request, use:

{
  "model": "openai/gpt-5",
  "session_id": "support-session-42",
  "routing": {
    "session_affinity": false
  }
}

BYOK considerations

If you bring your own provider key, that provider’s limits and policies apply. Fallbacks are still attempted where possible, but upstream account limits can constrain available options.

What to log

For production workloads, log request ids, response status codes, and model ids so you can correlate failures and confirm routing behavior when debugging.

Overview

Models & Routing

Features

Integrations

Workspace & Access

Reliability & Operations

Platform & Research

Migration Guides

Community

Routing and Fallbacks

Choose a routing mode

Model routing suffixes

How routing works at a high level

Choose an exact provider in the model id

Control routing and fallbacks

Presets constrain the fallback pool

Routing mode changes provider ranking

BYOK fallback is explicit

Dynamic routes attach policy to API keys

Fallback behavior

Cache-aware and session-aware affinity

BYOK considerations

What to log

​Choose a routing mode

​Model routing suffixes

​How routing works at a high level

​Choose an exact provider in the model id

​Control routing and fallbacks

​Presets constrain the fallback pool

​Routing mode changes provider ranking

​BYOK fallback is explicit

​Dynamic routes attach policy to API keys

​Fallback behavior

​Cache-aware and session-aware affinity

​BYOK considerations

​What to log

​Related guides

Choose a routing mode

Model routing suffixes

How routing works at a high level

Choose an exact provider in the model id

Control routing and fallbacks

Presets constrain the fallback pool

Routing mode changes provider ranking

BYOK fallback is explicit

Dynamic routes attach policy to API keys

Fallback behavior

Cache-aware and session-aware affinity

BYOK considerations

What to log

Related guides