ModelRefs / Batch Inference — AI Glossary
Batch Inference — AI Glossary
Processing a large volume of model requests offline in bulk, trading latency for cost and throughput efficiency.
Overview
Batch inference jobs run without real-time latency requirements, enabling larger batch sizes and lower GPU idle time. OpenAI Batch API, Anthropic Message Batches, and AWS Bedrock Batch offer ~50% cost reduction vs. real-time. Ideal for embedding generation, classification at scale, and offline eval runs.
Reference details
| Topic | inference |
|---|---|
| Also known as | batch processing, offline inference |
| Last reviewed | 2026-06-24 |
Related terms
How it works
- Requests are submitted as a file of independent jobs rather than as individual calls, so the provider receives the whole workload before scheduling any of it.
- The provider commits to a completion window instead of a per-request latency, which frees the scheduler to place work wherever capacity is idle.
- That freedom is what the discount pays for: batch jobs fill the troughs left by interactive traffic, so the marginal GPU-hour they consume would otherwise have been wasted.
- Larger batch sizes become available because no request is waiting on a user, and larger batches amortise each weight read across more sequences.
- Results are written back as a file keyed by the caller's own request identifiers, so reassembly is a join rather than a sequence of callbacks.
- Failures are per-request rather than per-job, which means a malformed entry returns an error beside the successful results instead of failing the submission.
Example: When the slow path is also the fast path
Suppose 2,000,000 documents need classifying at 800 input and 50 output tokens each, so 1,600,000,000 input tokens and 100,000,000 output tokens. Priced in real time at $3 per million input and $15 per million output, that is 1,600 x $3 = $4,800 plus 100 x $15 = $1,500, so $6,300. At the typical 50% batch discount the same work is $3,150, a saving of $3,150. The latency comparison is the part people get backwards. Assume the real-time path sustains 20 requests per second against its rate limit: 2,000,000 / 20 = 100,000 seconds, which is 100,000 / 3600 ≈ 27.8 hours. So the interactive path finishes this corpus in about 27.8 hours while a 24-hour batch window finishes it in at most 24 — the batch route is both cheaper and, for the corpus as a whole, sooner. The real-time path only wins when the unit of value is a single document rather than the corpus: the first classification arrives in a second or so interactively and not until the window closes in batch. That is the actual trade, and it is about which document matters rather than about speed.
Commonly confused with
Batch inference is not the same as the batch size a server uses internally, and the two are easy to conflate because both improve throughput by grouping work. Server-side batching groups whatever concurrent requests happen to be in flight and is invisible to the caller, who still gets an interactive response; batch inference is an API mode where the caller surrenders latency in exchange for a lower price. The second confusion is with asynchronous APIs, which return a job handle and complete in seconds to minutes at standard pricing — the distinguishing feature of a batch tier is the priced completion window, often 24 hours, not the fact that the call returns a handle.
When to use it
Reach for it when:
- For embedding or classifying a corpus, where no single item has a deadline and the whole set does.
- On offline eval runs, which are the canonical case — large, repeatable, and never user-facing.
- When the real-time path would be rate-limited anyway, since the serial interactive route took about 27.8 hours against a 24-hour window in the case above.
- For backfills and migrations, where halving the cost of a one-off bulk job is the entire decision.
Reach for something else when:
- Anywhere a user is waiting, which the completion window makes untenable regardless of average turnaround.
- For work whose inputs depend on earlier outputs, since the jobs in a submission are independent by construction.
- On small volumes, where the submission and reassembly machinery costs more engineering time than the discount returns.
- When results are needed within a window shorter than the provider's commitment, which is a hard constraint rather than a typical case.
Referenced by
This term is used by the following ModelRefs references:
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Batch Inference — AI Glossary.
Frequently asked questions
What is Batch Inference?
Processing a large volume of model requests offline in bulk, trading latency for cost and throughput efficiency.
Is Batch Inference the same as batch processing?
Yes — batch processing, offline inference are common aliases for Batch Inference.
What concepts are related to Batch Inference?
Closely related concepts include throughput, inference cost, eval.