LLMOps — AI Glossary

Reference details

Related terms

• Eval (Production Evaluation)
• Distributed Tracing (LLM)
• Prompt Management
• LLM Observability

How it works

1. Prompts are versioned as artefacts with their own identifiers, because a prompt change alters behaviour exactly as a weight change would and must be as revertible.
2. Every production call is traced: inputs, retrieved context, tool calls, output, token counts and latency, linked by a request id so a complaint can be reconstructed.
3. Evaluation runs against a dataset of saved cases rather than a single accuracy figure, scoring with graders or an LLM judge whose own agreement with humans is measured.
4. Regression gates sit in the deploy path: a prompt or model change runs the dataset first, and a drop below threshold blocks the release.
5. Cost and latency are monitored per request and attributed to a model, user or feature, because spend here is per call rather than per training run and drifts with traffic.

Example: Why the unit of cost changes the discipline

A training run costs a fixed amount once. An inference feature costs per call: 50,000 calls a day at 3,000 tokens and a blended rate of $5 per million tokens is 50,000 x 3,000 = 150M tokens, or $750 a day — about $22,500 a month. A prompt edit that adds 500 tokens of instructions raises it to 175M tokens and roughly $875 a day, a $3,750 monthly increase from a text change nobody treated as a deployment. That asymmetry is the reason prompt versioning and cost attribution belong in the same discipline.

Commonly confused with

The same discipline as MLOps applied where the model is called rather than trained, which moves the hard parts. Versioning covers prompts and retrieval corpora, not just weights; evaluation has no single accuracy number; and cost is per request rather than per training run. Observability is one component of it, not a synonym.

When to use it

Reach for it when:

• A model-backed feature is in production and someone has to answer why a specific response was wrong.
• Prompts, retrieval corpora or model versions change often enough that regressions need a gate rather than a reviewer.
• Spend scales with traffic and has to be attributable to a feature or customer.

Reach for something else when:

• Prototyping, where tracing and regression suites cost more than the iteration speed they protect.
• A single static prompt behind a low-traffic internal tool, where logging is enough.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to LLMOps — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path