Inference Cost — AI Glossary

Reference details

Related terms

• Token
• Latency
• Throughput

How it works

1. Input and output are priced differently because they are computed differently: the whole prompt is processed in one parallel pass, while each output token needs its own pass over the stored state.
2. That asymmetry is physical rather than commercial — a thousand input tokens cost one forward pass between them, and a thousand output tokens cost a thousand.
3. Prompt caching attacks the input term only, by retaining the computed state for a stable prefix so a repeated system block is charged at a reduced rate after its first use.
4. The cache is keyed on an exact prefix match, so a single changed character early in the prompt invalidates everything after it and the discount silently disappears.
5. Reasoning tokens are billed as output despite never being shown, which means the visible response length can understate the bill by a large multiple.
6. Batch tiers discount both terms in exchange for a completion window, so the levers are: shorten output, stabilise the prefix, and surrender latency where nobody is waiting.

Example: Where the bill actually comes from

A summarisation call sending 8,000 tokens and returning 500, at $3 per million input and $15 per million output: input costs 8000 × 3 / 1e6 = $0.024, output costs 500 × 15 / 1e6 = $0.0075. Input dominates here. Flip it — a short prompt generating 4,000 tokens — and output dominates. Which side to optimise depends entirely on the shape of your traffic.

input : 8,000 x $3  / 1e6 = $0.0240
output:   500 x $15 / 1e6 = $0.0075
total                      = $0.0315 per call

Commonly confused with

Token price is not cost per task. A cheaper model that needs three attempts, longer prompts or a retry on malformed output can cost more per completed task than a pricier one that succeeds first time. Compare cost per successful outcome, not cost per million tokens.

When to use it

Reach for it when:

• Sizing unit economics before committing to an architecture
• Deciding between routing tiers, caching or a smaller model
• Reasoning models are in play, where hidden reasoning tokens are billed

Reach for something else when:

• As the only axis — latency and quality usually matter more to the product
• Estimated from list price alone, ignoring retries, failures and caching

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Inference Cost — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path