Stress Testing (LLMs) — AI Glossary

Reference details

Related terms

• Load Testing (LLMs)
• Adversarial Attack
• Evaluation Harness

How it works

1. The input space is characterised by its boundaries rather than its typical cases: maximum context, empty input, malformed structure, unexpected language, adversarial instruction.
2. Context-length probes place known facts at controlled depths in a filled window and query for them, so retrieval failure is separable from reasoning failure.
3. Adversarial probes supply instruction-shaped text through every channel the system ingests — retrieved documents, tool results, file contents — rather than only through the user turn.
4. Schema probes feed tools malformed arguments and oversized results, since an agent's failure mode is usually in handling a tool's output rather than in choosing the tool.
5. Each probe asserts a specific observable property — the fact was recovered, the injected instruction was not followed, the malformed result did not crash the loop — rather than a quality impression.
6. Results are reported as a degradation curve against the stressed dimension, because the useful output is the point at which behaviour changes rather than a pass or fail.

Example: Half the stress-test budget is one data point

Suppose context behaviour is probed at eight lengths, doubling from 4k to 512k tokens: 4, 8, 16, 32, 64, 128, 256 and 512 thousand. Those sum to 1,020,000 tokens for one probe at each length, and running 20 probes per length to get a usable estimate is 20 x 1,020,000 = 20,400,000 input tokens, which at $3 per million is $61.20. That total is affordable, and its shape is the planning insight. The 512k point alone accounts for 20 x 512,000 = 10,240,000 tokens, which is 10,240,000 / 20,400,000 ≈ 50.2% of the entire budget — one of eight data points consuming slightly more than half the spend, because a doubling sequence puts most of its mass in the final term. Two consequences follow. First, dropping the longest point halves the cost and removes exactly the measurement most likely to reveal a problem, so it is the wrong economy. Second, if the probe count needs raising for statistical confidence, raising it only at the long end is nearly as expensive as raising it everywhere: going from 20 to 40 probes at 512k alone adds $30.72, against $61.20 for doubling every length. Budgeting a stress test therefore means deciding the long-end probe count first and treating the short lengths as free.

Commonly confused with

Stress testing is not load testing, and the two measure different systems. Load testing increases concurrent request volume to find where throughput saturates and latency degrades — a property of the serving infrastructure. Stress testing increases the difficulty of individual inputs to find where model behaviour degrades — a property of the model and the prompt. A deployment can pass load testing at ten thousand requests per second and fail stress testing on the first adversarial document it retrieves. It is also distinct from red teaming, though they overlap on adversarial inputs: red teaming is open-ended and human-driven, searching for failures nobody anticipated, while stress testing is a repeatable suite asserting known boundaries, which is what makes it suitable for regression on every model change.

When to use it

Reach for it when:

• Before any production deployment that ingests external text, where injection through retrieval and tool results is the live risk.
• When adopting a longer context window, since advertised length and usable length are different quantities.
• As a regression suite across model upgrades, where the value is detecting that a boundary moved.
• On agent workflows specifically, whose failure modes concentrate in tool-result handling rather than tool selection.

Reach for something else when:

• As a substitute for load testing, which measures infrastructure saturation rather than behavioural degradation.
• As a substitute for red teaming, since a fixed suite cannot find a failure mode nobody thought to encode.
• With the longest context point dropped to save money, which removes about half the cost and most of the diagnostic value.
• Reported as a pass or fail, where the informative result is the length or difficulty at which behaviour changed.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Stress Testing (LLMs) — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path