ModelRefs / Cold Start — AI Glossary

Cold Start — AI Glossary

The latency spike when an inference container or GPU worker boots from idle, loading model weights from storage into VRAM.

Overview

Cold starts range from 1 s (cached container image) to 30+ s (loading 70B weights from S3). Mitigated by warm pools (keeping a minimum of idle replicas), model caching on local NVMe, and smaller models. Critical consideration for real-time user-facing applications using serverless inference.

Reference details

Topic	inference
Last reviewed	2026-06-24

How it works

A new replica must acquire a container image, start a runtime, load model weights into memory, and allocate accelerator state before it can serve.
Weight loading dominates: the bytes moved scale with model size and the source matters enormously — local storage against object storage differ by an order of magnitude.
Image caching removes the first stage but not the weight transfer, which is why a cached container still cold-starts slowly for a large model.
Warm pools keep a minimum of idle replicas so scale-up is not on the request path, trading idle cost for latency.
Autoscaling that reacts to queue depth will always serve some requests through a cold start, so the policy decides how many rather than whether.

Example: Where the seconds actually go

A 70B model at 4-bit is roughly 35GB. Loading from object storage at 1GB/s takes about 35 seconds; from local NVMe at 5GB/s, about 7. Container start might be 1 second either way. So the same model cold-starts in 8 seconds or 36 depending only on where the weights live — the runtime was never the problem. This is why warm pools and local weight caches are the two levers, and why smaller models are disproportionately easier to scale.

Commonly confused with

Used for two different latencies that happen to share a name. This is infrastructure cold start: a replica is not ready and the request waits for it. The recommender-systems sense — a new user or item with no interaction history — is unrelated and appears in the same documents when a product has both. It is also not time-to-first-token, which measures prefill on a warm replica; conflating them produces latency budgets that look achievable until the first scale-up event.

When to use it

Reach for it when:

Capacity planning for bursty traffic where scale-up lands on the request path.
Deciding between warm pools and on-demand scaling, which is a cost-versus-latency trade.
Explaining why a large model is harder to autoscale than a small one.

Reach for something else when:

Describing prefill latency on a warm replica, which is time-to-first-token.
The recommender sense of a new user or item, which is a different problem.
Steady predictable traffic, where replicas stay warm and the concern does not arise.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Cold Start — AI Glossary.

Frequently asked questions

What is Cold Start?