Rate Limit Error — AI Glossary

Reference details

Related terms

• Retry with Exponential Backoff
• Tokens Per Minute (TPM)
• Requests Per Minute (RPM)

How it works

1. The provider meters each request against several independent counters — requests per minute, input and output tokens per minute, sometimes concurrent requests and a daily cap — scoped to an organisation, project or key.
2. Token-based counters must estimate before the response exists, so admission is decided against requested input plus declared maximum output. Declaring a large max_tokens consumes budget you never use.
3. When any counter is exhausted the request is rejected with HTTP 429 before inference begins, which is why a 429 costs nothing and arrives quickly.
4. Response headers carry the state: remaining requests and tokens, and reset times per counter. Retry-After, when present, is the provider telling you the wait — guessing is strictly worse than reading it.
5. Correct backoff is exponential with jitter. Without jitter, every client rejected in the same window retries in the same later window, and the thundering herd reproduces the overload it was meant to relieve.
6. Sustained 429s are a capacity signal, not a transient one. Retrying harder cannot create quota; shaping demand with a client-side limiter or raising the quota are the only two real responses.

Example: Why jitter is not a refinement

Suppose 200 concurrent workers are rejected inside the same second. With plain exponential backoff at 1s, 2s, 4s, all 200 retry at the same instant, and the burst is identical in size to the one that caused the rejection: the limit is breached again, and the pattern repeats with a longer period. Add full jitter — sleep uniformly in [0, 2^n) — and the first wave spreads over 1 second, the second over 2, the third over 4, giving expected peaks near 200, 100 and 50 requests per second. If the account's ceiling is 60 requests per minute, that is 1 per second, so even the jittered third wave at 50 per second is roughly 50x the limit, and backoff alone is the wrong tool. A client-side limiter admitting 1 request per second drains all 200 in 200 seconds with zero rejections, whereas the retry loop spends comparable wall-clock time generating hundreds of 429s and never establishes a stable rate. Backoff handles a transient spike; a limiter handles a quota.

Commonly confused with

A 429 is not a 503 and not a 500, and treating them alike produces either data loss or an outage. A 429 means your request was refused by policy before any work happened — it is safe to retry, and the provider has told you when. A 503 means the service could not serve it, which may or may not be safe to retry depending on idempotency. A 500 may mean the request was partially processed. The second confusion is between 429 and errors about context length: the former is quota over time, the latter is a single request exceeding the model's window, and no amount of backoff fixes the latter.

When to use it

Reach for it when:

• As the signal to engage a client-side rate limiter rather than simply retrying, once 429s are sustained rather than bursty.
• To decide which quota binds — compare remaining-requests against remaining-tokens headers, since chat traffic usually hits RPM and document traffic TPM.
• As the trigger for queue-based dispatch, where a bounded worker pool paces submissions to the known quota.
• As capacity-planning evidence when requesting a quota increase: 429 rate against quota is the argument.

Reach for something else when:

• As a condition to retry indefinitely. Without a retry ceiling and a dead-letter path, a sustained quota shortfall turns into unbounded latency for every caller.
• As a reason to ignore Retry-After in favour of your own schedule, which is how clients get throttled harder.
• Backed off without jitter in any multi-worker system, where synchronised retries recreate the overload.
• Confused with context-window errors: a request larger than the model's window fails identically on every attempt.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Rate Limit Error — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path