Token Budget — AI Glossary

Reference details

Related terms

• Inference Cost
• Token
• Streaming (Token Streaming)

How it works

1. A per-request maximum caps how many tokens the model may generate, which bounds both the bill and the worst-case latency for that call.
2. The declared maximum is also an admission quantity: providers meter against requested input plus declared output, so a large maximum consumes rate-limit headroom whether or not it is used.
3. Serving systems reserve key-value cache for the declared maximum in some configurations, so an over-generous value reduces the batch size the server can sustain.
4. Reasoning-capable models expose a separate budget for internal reasoning tokens, which are billed as output and do not appear in the response — so the two budgets control different quantities.
5. A budget stated in the prompt is a different mechanism entirely: asking for at most 200 words is an instruction the model may miss, while the parameter is enforced by the runtime.
6. Hitting the cap truncates rather than concludes, so the finish reason is the only signal distinguishing a complete response from one cut off mid-structure.

Example: Why a generous maximum costs throughput you never used

Take a 200,000 token-per-minute limit and a workload whose prompts are about 1,000 tokens. Declaring a 4,000-token maximum means each request is metered at 1,000 + 4,000 = 5,000, so admission allows 200,000 / 5,000 = 40 requests per minute. Declaring 200 — enough for the short answers this workload actually produces — meters each request at 1,000 + 200 = 1,200, allowing 200,000 / 1,200 = 167 requests per minute. That is 167 / 40 = 4.2x the throughput from changing one parameter, with no change to what the model generates, because the unused 3,800 tokens of headroom were being charged against the quota on every call. The cost side is unchanged either way — you pay for tokens produced, not declared — so this is purely a throughput effect, and it is invisible in a bill. It shows up as 429 responses at a request rate the quota should have supported, which is usually diagnosed as needing a quota increase.

Commonly confused with

A token budget is not a cost control in the way an iteration cap is not: it bounds one call and says nothing about how many calls are made, so an agent loop with a tight per-call budget and no loop budget is unbounded in aggregate. The separate confusion is between the output maximum and a reasoning budget on models that expose both. The first caps what the user sees; the second caps internal reasoning that is billed as output and invisible in the response, so a tight output maximum on a reasoning model can truncate the answer while the expensive part has already been paid for.

When to use it

Reach for it when:

• Every production call, set from the observed output length distribution rather than left at a default.
• Where rate-limit headroom matters, since the declared maximum is metered whether used or not.
• Separately from a reasoning budget on models exposing both, as they control different quantities.
• Checked alongside the finish reason, which is what distinguishes a complete response from a truncated one.

Reach for something else when:

• Left generously high by default, which silently consumes quota and reduces achievable request throughput.
• As an aggregate cost control, which needs a budget across calls rather than within one.
• Set so tight that legitimate responses truncate, where the failure presents as poor quality rather than as a limit.
• Confused with a prompt instruction to be brief, which the model may ignore and the runtime does not enforce.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Token Budget — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path