Cost Per Request — AI Glossary

Reference details

Related terms

• Cost Per Token
• Inference Cost
• Tokens Per Minute (TPM)

How it works

1. Input and output tokens are counted per request and priced at their separate rates, then summed — so the unit includes the request's actual shape.
2. Everything the request consumed is attributed to it: retrieval embedding calls, reranking, retries, and any agent steps.
3. That makes it comparable across differently shaped workloads in a way a per-token rate is not.
4. Attribution requires request-level tracing, since the cost of a multi-step request cannot be reconstructed from provider invoices alone.
5. Averages hide the distribution, so the useful figures are the median and a high percentile — a few expensive requests can dominate the bill.

Example: Why the mean is the wrong number to budget on

Ninety-five requests cost 1 unit each and five agentic ones cost 40: the total is 95 + 200 = 295 units over 100 requests, so a mean of 2.95 while the median is 1. Budgeting on the mean understates the simple traffic threefold and understates the expensive tail by more than tenfold. The 5% of requests consuming 68% of the spend is where any optimisation should go, and the mean is precisely the statistic that conceals them.

Commonly confused with

Treated as a restatement of cost per token, which it contains rather than repeats. A per-token rate is a price; this is a measured outcome that depends on prompt length, output length, retries and the number of steps a request took. Two systems on identical token prices can differ severalfold here. It is also not cost per task: a task may span several requests, and for an agentic workload that is the figure that matters commercially — this is the building block for it rather than the answer.

When to use it

Reach for it when:

• Comparing the real cost of differently shaped workloads on the same price list.
• Finding which requests dominate spend, which requires the distribution rather than the mean.
• Attributing cost to features or customers, which needs request-level tracing.

Reach for something else when:

• As a synonym for the provider's per-token rate, which is an input not an outcome.
• Reported as a mean, which hides the expensive tail that matters.
• For multi-request tasks, where cost per completed task is the commercially relevant unit.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Cost Per Request — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path