Serverless Inference — AI Glossary

Reference details

Related terms

• Cold Start
• Warm Pool
• Inference Provider

How it works

1. No accelerator is held while the endpoint is idle, which is the property the billing model rests on and also the cause of everything else in the sequence.
2. A request with no warm instance triggers provisioning: an accelerator is scheduled, the container image pulled, the weights read from storage into device memory, and the runtime initialised.
3. Weight loading dominates that path because the volume is gigabytes, so cold-start duration tracks model size far more closely than it tracks anything about the request.
4. Once warm the instance serves normally and is retained for a keep-alive window, so subsequent requests inside that window pay nothing of the start-up cost.
5. Concurrency beyond one instance's capacity adds replicas, each of which pays its own cold start, which is why a traffic spike produces slow requests rather than merely queued ones.
6. Providers reduce but cannot remove this by holding pre-initialised capacity, and that capacity is reserved rather than free — which is the sense in which scaling genuinely to zero and never cold-starting are mutually exclusive.

Example: Where the break-even sits, and how to compute yours

A dedicated endpoint bills continuously whether or not traffic arrives; serverless bills per token but adds a cold start of seconds when scaled to zero. For bursty or low-volume workloads serverless is far cheaper. For steady high traffic the dedicated endpoint wins on both cost and latency — the crossover is a traffic-shape question, not a preference, and it is a single division. Price the dedicated side first: one accelerator at $2.50 an hour held continuously is 2.50 x 24 = $60 a day regardless of traffic. If serverless costs about $0.0015 per request for the same work, the two meet at 60 / 0.0015 = 40,000 requests a day. Below that the dedicated endpoint is paying for idle capacity and serverless wins; above it the per-request price compounds past the fixed cost. The number moves with request size and hourly rate, which is why it is worth recomputing rather than inheriting — but the shape of the decision is always this division.

Commonly confused with

Serverless does not mean no cold start. Scaling to zero is precisely what creates one: multi-gigabyte weights have to be loaded before the first token. Providers mitigate it with warm pools, but the trade-off is inherent to scaling to zero. It is also not autoscaling: an autoscaled dedicated deployment keeps a floor of running instances, so it has no cold start on the common path and no scale-to-zero saving either, while serverless is specifically the version where the floor is zero.

When to use it

Reach for it when:

• Traffic is bursty, unpredictable or low-volume
• Idle GPU cost is the dominant line item
• A delay on the first request after an idle period is acceptable
• Below the break-even request volume, which is a single division away from your own rates

Reach for something else when:

• Interactive products where a cold start lands on a real user
• Steady high throughput, where dedicated capacity is cheaper per token
• Strict tail-latency commitments, which a cold start violates by definition
• Treating a warm-pool offering as scale-to-zero, which is a dedicated endpoint priced differently

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Serverless Inference — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path