Load Testing (LLMs) — AI Glossary

Reference details

Related terms

• P99 Latency
• Tokens Per Second (TPS)
• Throughput

How it works

1. A traffic model is defined first: arrival rate, concurrency, and the prompt and output length distribution. Token lengths matter more than request counts, because an LLM's cost and latency scale with tokens rather than calls.
2. Load is generated against the real serving stack — the gateway, the queue, the engine — since the queueing behaviour under contention is usually what fails and it does not appear when the engine is tested alone.
3. Time-to-first-token and inter-token latency are measured separately, because they have different causes: the first reflects queueing and prefill, the rest reflects decode throughput and batch composition.
4. Results are read as distributions rather than averages. A mean hides the tail, and the tail is what users experience during the moments that matter.
5. Load is raised until something breaks, and the breaking mode is the finding — queue growth, rate-limit rejections, out-of-memory on long contexts, or latency degrading while throughput holds.
6. The run must be long enough for autoscaling and cold starts to participate, since a short test measures warm steady state and misses the behaviour that makes a launch fail.

Example: Why a percentile needs a sample size, and why tails multiply

A P99 is estimated from the slowest 1% of observations, so at n = 100 requests it rests on 100 / 100 = 1 sample — a single draw, with no resolution at all. At n = 10,000 it rests on 100 samples, which is enough for the figure to be stable. Testing a P99 claim with a few hundred requests is therefore not a weak measurement but an absent one. The second piece of arithmetic is why the tail matters more than its share suggests. Suppose a page makes 3 sequential model calls and each independently has a 1% chance of landing in its slow tail. The probability that none does is 0.99^3 = 0.970, so 1 - 0.970 = 3.0% of page loads contain a tail call — triple the per-call rate. At 5 calls it is 1 - 0.99^5 = 4.9%, and at 10 calls 1 - 0.99^10 = 9.6%, so nearly one request in ten hits a tail somewhere. This is why per-call P99 is the wrong target for a multi-call feature: the user's experience is the tail of the composition, not of any one call.

Commonly confused with

Load testing is not benchmarking, and conflating them produces numbers that describe nobody's system. A benchmark measures a model's capability on fixed tasks under controlled conditions. Load testing measures a deployment's behaviour under traffic, so its results are properties of your gateway, quotas, batch settings and hardware rather than of the model. A published tokens-per-second figure is a benchmark; what your stack does at your concurrency with your prompt lengths is only discoverable by testing it.

When to use it

Reach for it when:

• Before launching a high-traffic feature, where the purpose is finding the breaking mode rather than confirming a number.
• Capacity planning against a quota, where the question is how much headroom exists before 429s start.
• After a configuration change to batching, concurrency or model, each of which moves the latency distribution.
• Validating autoscaling and cold-start behaviour, which only appear in runs long enough to trigger them.

Reach for something else when:

• With too few requests to estimate the percentile you are claiming — a P99 from a few hundred requests is one or two samples.
• Against a staging environment with different quotas or hardware, which measures a system you will not run.
• With uniform prompt lengths, which misses the long-context failures that production traffic finds.
• Reported as a mean. The average is the one statistic that cannot tell you whether the system is acceptable.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Load Testing (LLMs) — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path