Use this workflow to choose quantization variants safely.
To enforce the choice in Phaseo Gateway, send an OpenRouter-compatible routing filter:
The filter applies to provider-model offers, not just provider names. Matching is case-insensitive and supports unambiguous aliases such as float8 for FP8 and bfloat16 for BF16. An offer with missing quantization metadata does not qualify. If no eligible offer matches, the request fails with routing diagnostics; Phaseo does not fall back to an unverified quantization. The equivalent routing.quantizations shape is also accepted.
Step 1: define your success criteria
Before choosing a quant:
- Set quality thresholds (task accuracy, pass rates, structured output validity).
- Set latency and cost targets.
- Define unacceptable failure modes.
Step 2: pick a high-quality baseline
Start with a stronger baseline (often BF16 or FP16) and measure:
- Output quality
- Median and tail latency
- Token/cost profile
This becomes your comparison anchor.
Step 3: step down gradually
Evaluate progressively lower precision:
FP8 or INT8FP4 or INT4 only if needed for cost/throughput
Do not skip directly to the smallest variant unless you already have strong evidence it works for your workload.
Step 4: evaluate real workloads
Use production-like prompts:
- Long prompts
- Multi-step reasoning
- Tool calling
- Edge cases from support tickets
Step 5: decide per endpoint/use case
One quantization does not need to fit everything.
Examples:
- Customer support and critical decisioning: higher precision.
- Large-volume classification/summarization: lower precision can be acceptable.
Suggested starting points
Deployment policy
- Keep a fallback model/quantization configured.
- Monitor quality drift after rollout.
- Re-test when provider runtime or model revision changes.
Last modified on August 23, 2026
