ModelRefs / P99 Latency — AI Glossary

P99 Latency — AI Glossary

The 99th-percentile request latency; the worst-case response time experienced by 1-in-100 requests under load. The average hides what users actually experience.

Overview

P99 (and P95, P50) latency percentiles capture tail behavior that median or average masks. For LLM APIs, P99 spikes during traffic bursts, KV cache evictions, or cold starts. SLOs typically specify P50 < 300 ms TTFT and P99 < 2 s. Profiling with tools like Grafana or Datadog exposes P99 regressions.

Reference details

Topicinference
Also known astail latency, 99th percentile latency
Last reviewed2026-06-24

Related terms

How it works

  1. Requests are measured individually and the distribution is sorted, so a percentile is a position in that order rather than a computed average.
  2. P99 is the value 99% of requests come in under, which means it is defined by the slowest 1% and is insensitive to improvements in the fast majority.
  3. Percentiles are computed per window, so a short window gives a noisy figure and a long one hides a transient spike.
  4. They cannot be averaged across services: combining two services' P99s does not give the P99 of the pair, because the slow requests need not coincide.
  5. For streaming responses the useful percentile is on time-to-first-token and on inter-token gaps separately, since one long pause is invisible in total duration.

Example: Why improving the median can leave users worse off

A thousand requests: 990 at 200ms and 10 at 5,000ms. The mean is (990 x 200 + 10 x 5000) / 1000 = (198,000 + 50,000) / 1000 = 248ms, which looks healthy, and the median is 200ms. P99 is 5,000ms. Halving the fast path to 100ms moves the mean to 149ms and the median to 100ms while P99 does not budge — the 10 users in the tail see no change at all. Every number except the one that describes them improved.

Commonly confused with

The average hides what users actually experience. A one-in-a-hundred worst case sounds rare until a page makes several model calls: at ten calls, the chance of at least one landing in that tail is 1 − 0.99¹⁰ ≈ 9.6%, so roughly one page view in ten is as slow as the P99. Report a percentile and a sample size; an average alone is not a latency claim.

When to use it

Reach for it when:

  • Setting an SLO that reflects what the worst-served users experience.
  • A page makes several model calls, where one slow call dominates perceived latency.
  • Detecting regressions from cache eviction, cold starts or burst contention, which averages absorb.

Reach for something else when:

  • Capacity planning, where mean throughput is the relevant quantity.
  • Sample sizes too small for the 99th percentile to be a stable estimate.
  • Combining percentiles across services as though they added.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to P99 Latency — AI Glossary.

Frequently asked questions

What is P99 Latency?

The 99th-percentile request latency; the worst-case response time experienced by 1-in-100 requests under load.

Is P99 Latency the same as tail latency?

Yes — tail latency, 99th percentile latency are common aliases for P99 Latency.

What concepts are related to P99 Latency?

Closely related concepts include time to first token, tokens per second, throughput.