Rate Limiting — AI Glossary

Reference details

Related terms

• Model Gateway (LLM Gateway)
• Inference Cost

How it works

1. Limits are implemented as token buckets rather than fixed windows: a bucket of some depth refills continuously at the stated rate, and a request consumes from it.
2. That means a burst up to the bucket's depth is allowed after an idle period, while the sustained ceiling is the refill rate — so a limit stated per minute does not mean the quota resets on the minute.
3. A request consumes against the token budget at admission using the maximum it could produce, since the actual output length is not yet known.
4. So a generous maximum reserves capacity the request may never use, and lowering it raises effective concurrency without any change to the quota.
5. The reservation is reconciled against actual usage after the response completes, which is why observed throughput can exceed what the admission arithmetic appeared to allow.
6. Both the request and token counters are enforced, and the binding one is whichever is exhausted first — which moves as prompt length changes rather than staying fixed.

Example: Work out which limit actually binds

Say an account is capped at 500 requests per minute and 200,000 tokens per minute. If an average request carries 2,000 tokens of prompt and response, the token cap allows 200,000 / 2,000 = 100 requests per minute — one fifth of the request cap, which you will therefore never reach. Teams size concurrency against the RPM figure, see 429s at a fifth of it, and conclude the limit is wrong. Compute both, take the smaller, and remember that the binding constraint moves: shortening prompts raises your effective request ceiling without any change to the quota.

Commonly confused with

Provider rate limiting protects the provider; your own rate limiting protects you — from runaway spend, from a retry storm you generated, and from one tenant consuming another's capacity. They need separate handling. A 429 is also not an error in the usual sense: it is a scheduling signal, and the correct response is to wait and retry, not to fail the request.

When to use it

Reach for it when:

• Any production integration — assume limits and design the queue before you hit them
• Multi-tenant systems, where per-tenant budgets prevent one caller starving the rest
• Alongside backoff with jitter, so retries do not synchronise into a second spike

Reach for something else when:

• Treating 429 as a failure to surface to users instead of a wait-and-retry
• Sizing against the request cap alone when the token cap is the one that binds
• Spreading load across multiple keys to evade limits, which usually violates the terms

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Rate Limiting — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path