Como a Phaseo mede latência e throughput

Time to first token (TTFT) is measured from the gateway receiving a request to the first content-bearing generated output delivered by a streaming response. Provider TTFT starts at the selected provider dispatch. Metadata-only SSE frames do not stop either clock, and TTFT is not reported for non-streaming responses.

Provider duration runs from selected provider dispatch to the terminal response. Gateway end-to-end duration runs from gateway request start to completion. Phaseo overhead is their non-negative difference.

Throughput is output tokens divided by the full provider duration in seconds. Output speed excludes TTFT and divides the remaining output tokens by the remaining generation interval. TPOT and ITL are request-level averages over that same post-first-token interval.

Public performance views use rolling windows such as the last 24 hours for detailed performance and longer periods for leaderboard and trend summaries.

Public charts support P01, P05, P10, P25, P50, P75, P90, P95, and P99. P99 means the value at or below which 99 percent of observations fall. For latency, lower is better; for throughput, higher is better, so lower percentiles expose the slow tail.

Phaseo excludes obviously invalid records, unknown identifiers, and rows without enough usable request volume to support a meaningful comparison. Model pages can segment operational traffic by streaming mode, input-token context bucket, provider, and Cloudflare execution location.

Performance pages and leaderboards only rank rows with finite, positive throughput or latency values once the relevant thresholds are met.

Latency and throughput are operational measurements, not intrinsic constants of a model. They can move because of provider routing, regional load, model updates, queueing behavior, prompt length, output length, and transport conditions.

A model may therefore rank differently across providers, across time windows, or across the gateway and the provider's own direct benchmarks.

Public performance charts are designed for directional comparison. They are not a substitute for your own workload-specific benchmarking under your own prompt mix, concurrency, and latency budget.

If a page has insufficient current data, Phaseo may show an empty state and treat that route as a weak search candidate until enough public volume exists.