Phaseo Team2 नव॰ 2026पढ़ने में 6 मिनट लगेंगे
LLM rate limits and AI budgets for every key, person, team and end user
Rate limit LLM requests, tokens and concurrency per API key, team member, department or end user, and cap AI spend with department budgets that block at 100%.

Most teams find out they needed a limit right after something used up the shared capacity. An agent loops overnight, a batch script retries every failure at once, or one heavy user of your app slows things down for everyone else.
Phaseo now has rate limit policies and department budgets. You decide how many requests, tokens and parallel calls each API key, team member, department or end user gets, and how much each department can spend before its requests stop. It works across the models and providers you call through Phaseo. Limits for keys, members and departments need no code changes, and end-user limits need your app to send an end-user ID, if it doesn't already. Nothing is limited until you set it up.
Rate limits for keys, members, guardrails and end users are available on every workspace. Department limits and budgets come with Self Serve Enterprise, alongside SSO, SCIM and directory management.
Who needs LLM rate limits and AI budgets
Teams building AI products
Companies rolling out AI by department
Platform teams running agents
Whoever owns the AI bill
How rate limiting works on Phaseo
Every request passes through the same checks before it reaches a model. If any check fails, Phaseo returns a 429 straight away. The request never reaches the provider, so it costs nothing.
Request arrives
From your app, an agent or a teammate's key.For any model or provider
Rate limit policies
Requests, tokens and concurrency for the key, member, department, guardrail and end user.429 rate_limit_exceeded
Department budget
Daily, weekly or monthly spend for the departments the request counts toward.429 key_limit_exceeded
Model provider
Routing and fallbacks as usual.Response, with rate limit headers
Rate limit policies for requests, tokens and concurrency
A policy is a named set of limits that you can reuse anywhere:
| Limit | What it counts | Good for |
|---|---|---|
| Requests per minute | Calls over a rolling minute | Fair access and abuse protection |
| Tokens per minute | Input and output tokens over a rolling minute | Matching real model load and cost |
| Concurrent requests | Requests in progress at the same time | Long streams, agents and batch scripts |
Rate limit per API key, user, team or end user
You assign a policy to whatever it should limit. A department can share a pool of 600 requests a minute while each person in it is held to 60. When several policies apply, Phaseo counts the request against each of them, and the first one to run out refuses it.
keyAPI key
memberTeam member
departmentDepartment
guardrailGuardrail
end_user_defaultEnd user
user field or X-Phaseo-User-Id header.Fallbacks to another provider within one request count once, and responses served from cache don't count at all.
Exact limits when you need them
Request and token limits are counted in the background by default, so they add no latency. The trade-off is that a sudden burst can briefly go over the limit before the count catches up.
When a limit must hold exactly, mark the policy strict. Strict limits are checked before the request is sent, which adds a little latency. If the check can't complete within two seconds, Phaseo refuses the request instead of letting it through. Concurrency limits are always exact: a request holds its slot until its response has been sent in full, including the whole of a streamed response.
AI budgets per department that stop spending at 100%
On Self Serve Enterprise workspaces, give a department a daily, weekly or monthly budget, and Phaseo tracks its spend as requests complete. Owners and admins get a push notification at about 80% and again at 100%. Once a budget is used up, Phaseo refuses that department's API-key requests until the period resets, at midnight UTC, on Mondays or on the 1st of the month.
A department's spend includes the keys linked to it. Keys that aren't linked to a department count toward the departments of the person who created them, and OAuth requests count toward the departments of the signed-in person.
Count each person once with primary department attribution
People often belong to more than one department, especially when departments are synced from your identity provider through SCIM. By default their unlinked keys count toward every department they're in, so each of those departments sees the spend.
To count each person once, switch the workspace to primary department only under Settings → Notifications. Their spend then goes to their primary department alone. Department budgets and department rate limits follow the same setting, so the numbers agree everywhere. A key linked to a department, or a member's department override, still takes priority.
429 responses and rate limit headers
Responses to requests that a policy covers include the standard rate limit headers, so OpenAI-compatible clients and retry logic work unchanged:
x-ratelimit-limit-requests: 60
x-ratelimit-remaining-requests: 41
x-ratelimit-reset-requests: 60s
A refused request returns 429 with a Retry-After header and a body that names the limit that refused it:
{
"error": {
"code": "rate_limit_exceeded",
"reason": "policy_requests_per_minute",
"limit_scope": "department",
"policy_id": "…"
}
}
A department budget refusal uses key_limit_exceeded with the reason department_monthly_cost_budget_reached (or daily or weekly), the department's name and reset_at, so you can tell people when they can try again.
Set up rate limits and budgets in the dashboard or the API
Everything is in the dashboard under Settings → Rate limits and Settings → Budgets. To manage limits from code, use the management API. For example, a shared pool for a department with a monthly budget on top:
curl https://api.phaseo.app/v1/rate-limits/policies \
-H "Authorization: Bearer $PHASEO_MANAGEMENT_KEY" \
-H "Content-Type: application/json" \
-d '{"name": "Support team", "requests_per_minute": 600, "concurrent_requests": 20}'
curl https://api.phaseo.app/v1/rate-limits/assignments \
-H "Authorization: Bearer $PHASEO_MANAGEMENT_KEY" \
-H "Content-Type: application/json" \
-d '{"policy_id": "<policy-id>", "target_type": "department", "target_id": "<department-id>"}'
curl https://api.phaseo.app/v1/department-budgets \
-H "Authorization: Bearer $PHASEO_MANAGEMENT_KEY" \
-H "Content-Type: application/json" \
-d '{"department_id": "<department-id>", "interval": "monthly", "limit": 500}'
Rate limit and budget commands for the Phaseo CLI, and resources for the Terraform provider, come in their next releases. You'll then be able to review and version limits alongside the rest of your infrastructure.
Good to know
- These limits come on top of each workspace's built-in allowance, which still applies.
- By default, a policy lets requests through if its limits briefly can't be checked. A department budget does the same if its check is unavailable, so a department can briefly spend past its budget. Use a strict policy when a rate limit must never be exceeded.
- Batch jobs, realtime sessions and OAuth requests aren't blocked by department budgets yet. Their spend still counts toward the department.
- Budget notifications are based on usage that can lag by up to an hour, so a request can be refused before the 100% notification arrives.
- Policy changes apply within about a minute. Department membership changes, including those from SCIM, can take up to about five minutes.
Get started
Open Settings → Rate limits to create your first policy, or Settings → Budgets to give a department a budget. The rate limits guide covers every field, error reason and header, and guardrails let you restrict which models and providers each key can use.
अक्सर पूछे जाने वाले प्रश्न
LLM rate limiting caps how much traffic can reach a model in a given time, usually as requests per minute, tokens per minute or the number of requests in progress at once. It stops one caller from using up shared capacity or budget, so other callers' requests still get through.
Create a Phaseo rate limit policy and assign it as the end-user default. Every end user then gets their own allowance, identified by the user field or X-Phaseo-User-Id header on each request. Requests without an end-user ID aren't limited by that policy.
Requests per minute counts calls over a rolling minute. Tokens per minute counts input and output tokens over a rolling minute, so it tracks real model load. A concurrency limit caps how many requests can be in progress at the same time, which suits long streams and agents.
Yes, on Self Serve Enterprise workspaces, which include departments. Give a department a daily, weekly or monthly budget. Owners and admins get a push notification at about 80% and at 100%, and once the budget is used up, Phaseo refuses API-key requests that count toward that department until the period resets.
The request returns HTTP 429 with a Retry-After header. A rate limit returns the rate_limit_exceeded error code with the policy and scope that refused it. A department budget returns key_limit_exceeded with the department's name and when the budget resets.
Not by default. Request and token limits are counted in the background. Strict policies and concurrency limits are checked before the request is sent, which adds a little latency in exchange for exact limits.
Responses served from cache don't count. Fallbacks to another provider within one request count once.
Yes. Phaseo sends the standard x-ratelimit-limit-requests, x-ratelimit-remaining-requests and x-ratelimit-reset-requests headers, the matching -tokens headers for token limits, and a Retry-After header on 429 responses, which OpenAI-compatible clients and retry libraries already understand.