ModelRefs / LLM Observability — AI Glossary

LLM Observability — AI Glossary

Collecting traces, logs, and metrics across LLM calls to understand system behavior, diagnose failures, and measure output quality.

Overview

LLM observability instruments inputs, outputs, latency, token usage, tool calls, and retrieval quality for every request. Platforms: LangSmith, Langfuse, Weave, Arize Phoenix. Complements traditional APM tools which lack LLM-specific semantics.

Reference details

Topic	operations
Last reviewed	2026-06-24

How it works

Every request emits a trace with a stable id, so one user complaint can be located rather than inferred from aggregates.
Spans record the inputs and outputs of each step — retrieval, each model call, each tool — which is what makes a wrong answer attributable to a stage.
Token counts and latency are recorded per span, so cost and time have a cause rather than a total.
Traces are sampled or retained in full depending on volume and cost, and the sampling rate bounds what can be answered later.
Retained traces become evaluation cases, which is the step that turns observation into a regression gate rather than a dashboard.

Example: What the sampling rate costs you later

At 1% sampling and 100,000 requests a day, 1,000 traces are kept. A failure affecting 1 request in 10,000 occurs about 10 times a day and is captured about 0.1 times — so most days there is no example of it at all, and the bug is unreproducible from the data. Raising sampling to 10% captures about one a day. The storage is ten times larger and the class of question answerable changes completely, which is why the rate is a decision about which failures remain investigable.

Commonly confused with

Monitoring tells you a known metric moved; observability is being able to ask why afterwards, including questions you had not thought to ask in advance. Tracing is the mechanism that makes it possible for multi-step applications, by joining spans under one request. For LLM systems the spans carry prompts and outputs, which is why this is also a store of user data and needs the same handling.

When to use it

Reach for it when:

Production requests must be explainable after the fact, including questions not anticipated in advance.
A multi-step pipeline where a bad output needs attributing to a stage.
Cost and latency need attribution per feature, model or customer.

Reach for something else when:

A known metric just needs a threshold alert, which is monitoring and cheaper.
Volume is low enough that application logs answer the questions asked.
Sampling is too sparse to capture the failures being investigated, where the instrumentation gives false confidence.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to LLM Observability — AI Glossary.

Frequently asked questions

What is LLM Observability?