Tokens Per Minute (TPM) — AI Glossary

Reference details

Related terms

• Requests Per Minute (RPM)
• Rate Limiting
• Retry with Exponential Backoff

How it works

1. The provider meters tokens over a rolling window, counting prompt and completion together — so a long prompt consumes the same budget as a long answer.
2. A request is admitted only if its projected total fits the remaining budget, which means completion tokens are reserved against the maximum you allowed, not the number actually produced.
3. Exceeding the limit returns a 429 with a retry hint rather than queueing, so the caller owns the backpressure.
4. Limits are per key and per model and usually accompany a separate requests-per-minute ceiling, either of which can bind first.
5. Correct client behaviour is exponential backoff with jitter: fixed-interval retries from many workers re-synchronise into the next burst.

Example: Which ceiling actually binds

A workload sends a 2,000-token prompt and receives 500 tokens, so 2,500 tokens per call. Against a 40,000 TPM allowance that is 40,000 / 2,500 = 16 calls per minute. If the key's requests-per-minute limit is 60, TPM binds first and the extra request headroom is unusable. Trimming the prompt to 1,500 tokens makes each call 2,000 and allows 20 — a 25% throughput gain from the prompt alone, no infrastructure change. Reserving max_tokens at 2,000 instead of 500 would have reserved 4,000 per call and allowed only 10, which is why over-generous max_tokens quietly halves throughput.

Commonly confused with

Confused with throughput, which it constrains without measuring. Throughput is what a system achieves — tokens actually processed per unit time; TPM is an administrative ceiling a provider imposes on a key. Hitting a TPM limit produces 429s, not slow responses, which is why it presents as an availability problem rather than a performance one. It is also separate from requests-per-minute: a workload with small prompts usually exhausts RPM first and one with large prompts exhausts TPM, and diagnosing the wrong ceiling leads to optimising the wrong thing.

When to use it

Reach for it when:

• Capacity planning a workload against a provider's published tier limits.
• Diagnosing 429s under load and deciding whether to trim prompts, batch, or raise the tier.
• Explaining why reducing prompt size raises throughput without touching infrastructure.

Reach for something else when:

• Describing how fast the model generates, which is tokens per second on the serving side.
• Describing self-hosted capacity, where the ceiling is memory and bandwidth rather than a quota.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Tokens Per Minute (TPM) — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path