Braintrust — AI Glossary

Reference details

Related terms

• LangSmith
• Langfuse
• Experiment Tracking

How it works

1. Datasets of cases are the primary object, so an evaluation is a run over a named dataset rather than an ad-hoc script.
2. An experiment records a run with its prompt, model and scores, which makes two variants comparable on identical inputs.
3. Scorers are configurable per experiment, mixing deterministic checks with model-based judges.
4. Production traffic can be scored online with the same scorers, so the offline and live signals are the same measurement.
5. Failing production cases are promoted into datasets, which is the loop that turns incidents into permanent regression coverage.

Example: Why comparing on identical inputs is the point

Prompt A is tested on Monday against 80 cases and prompt B on Thursday against 95, after someone added cases. B scores higher. Nothing follows: the inputs differ, and the 15 new cases could be easier. Running both against the same named dataset makes the difference attributable to the prompt. The discipline is identical to holding a test set fixed, and the common failure is a dataset that grows between runs so every comparison is contaminated.

Commonly confused with

Sits in the same category as Langfuse, Phoenix and LangSmith, with its centre of gravity on evaluation rather than tracing — experiments, scoring and dataset iteration, with observability alongside. The practical question when choosing among these is not which has more features but which owns your evaluation datasets, since those are the asset that becomes expensive to move later.

When to use it

Reach for it when:

• Evaluation and experiment comparison are the centre of the workflow.
• The same scorers should run offline and against production traffic.
• Failing production cases need promoting into regression datasets.

Reach for something else when:

• Deep per-request tracing is the primary need, which observability platforms centre on.
• Self-hosting is a requirement.
• Datasets change between runs, which makes experiment comparison meaningless whatever the tool.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Braintrust — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path