Warm Pool — AI Glossary

Reference details

Related terms

• Cold Start
• Serverless Inference
• GPU Utilization (MFU)

How it works

1. A minimum replica count is declared, and the scheduler keeps that many workers alive with the model weights resident in GPU memory regardless of traffic.
2. Each warm worker has already paid the one-time costs — container start, runtime initialisation, weight load from storage into device memory, and any compilation or kernel autotuning — so an arriving request begins at the forward pass.
3. A load balancer routes to warm workers first. Requests beyond warm capacity either queue for a warm slot or trigger a cold worker, and which of the two happens is the single most important configuration choice.
4. Autoscaling adds workers above the floor on a demand signal, typically queue depth or utilisation, and scale-up is bounded below by the cold-start time the pool exists to hide.
5. A scale-down delay keeps workers past the point demand drops, because a worker released and immediately re-created costs a full cold start to recover.
6. The floor is sized against demand variance, not the mean. A pool sized to average load leaves every burst to be served cold, which is the common misconfiguration.

Example: What the idle floor actually costs, and what it buys

A 70B model at 4-bit needs roughly 35GB and takes about 40 seconds to cold start. Traffic is 20 requests per hour across a 10-hour business day, bursty, and near zero overnight. One warm replica held for the 10-hour day costs 10 GPU-hours a day; at 2 units per GPU-hour that is 20 units daily and about 600 a month. The 20 x 10 = 200 daily requests at 3 seconds of compute each consume 200 x 3 = 600 seconds, ten minutes of the 10 hours — so the replica is idle roughly 98% of the time and the 600 units a month is almost entirely paying to not wait. State the benefit in the same units: if bursts arrive 5 times a day, the pool avoids 5 x 40 = 200 seconds of visible latency per day, so each avoided second of first-token wait costs about 20 / 200 = 0.1 units. Whether that is worth paying is a product decision, but it is now a priced one rather than an assumption.

Commonly confused with

A warm pool is not autoscaling, and it is not a response cache. Autoscaling reacts to load after it arrives and is bounded by the cold-start time it cannot avoid; a warm pool pre-pays to remove that bound for a fixed amount of capacity. The two compose — the floor absorbs bursts, autoscaling handles sustained growth — and a system with only autoscaling has no answer for the first request of a burst. The cache confusion matters too: a warm pool holds weights and runtime state, not responses, so it reduces time-to-first-token and nothing about repeated-query cost.

When to use it

Reach for it when:

• Interactive, user-facing endpoints where a cold start is visible and a burst leader waiting tens of seconds is a defect.
• Bursty traffic with predictable shape — business hours, scheduled jobs — where the floor can follow a schedule instead of being held continuously.
• Large models, where cold start scales with weight size and the avoided wait is largest.
• Workloads with a latency SLO, since a warm floor is the only mechanism that bounds time-to-first-token for the first concurrent request.

Reach for something else when:

• Batch and asynchronous workloads, where a 40-second start is amortised over a long job and the idle floor is pure cost.
• Genuinely unpredictable low-volume traffic, where a floor large enough to catch bursts is idle almost all the time.
• As a fix for throughput problems. A warm pool removes startup latency; it does not add serving capacity beyond its replica count.
• Before measuring where the cold start goes. If the time is dominated by weight transfer from object storage, a faster weight source may cut it more cheaply than holding GPUs idle.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Warm Pool — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path