Experiment Tracking — AI Glossary

Reference details

Related terms

• MLflow
• Weights & Biases (W&B)
• Model Registry

How it works

1. The training script is instrumented with a few calls that open a run, which the tracker creates as a server-side record with its own identifier before any training happens.
2. Hyperparameters are captured at that moment rather than inferred later, because the configuration that produced a result is not recoverable from the result.
3. Metrics are logged per step or per epoch and streamed to the store as they are produced, so a crashed run still leaves everything it had reached.
4. The environment is captured automatically — git commit, dirty-tree state, package versions, hardware — since those are the fields nobody remembers to record by hand.
5. Checkpoints and other artifacts are uploaded and attached to the run id, which is what turns a file on a disk somewhere into something a result can be traced to.
6. Comparison is then a query over that store rather than an exercise in memory: runs are filtered and sorted by their recorded configuration, and the winning one carries everything needed to rebuild it.

Example: The run you cannot reproduce, and the odds against reconstructing it

A checkpoint from three weeks ago outperforms everything since and nobody can rebuild it: the learning rate is not recorded, the data snapshot moved, the commit is unknown. Tracking exists so that sentence is never true — hyperparameters, metrics, data version and commit captured at run time, because none of it can be reconstructed afterwards. The reason reconstruction fails is that it needs every field at once. Rebuilding a run requires the learning rate, the data snapshot, the commit, the random seed and the package versions. Suppose each is independently recoverable three weeks later with probability 0.7 — a generous estimate for anything held in memory or inferred from a chat log. All five together is 0.7^5 ≈ 0.168, so the run is reproducible about one time in six. Capturing them costs a few lines at run start; recovering them is a coin flip taken five times in a row.

Commonly confused with

Experiment tracking is not model monitoring. Tracking records training runs before deployment; monitoring watches behaviour after it. Both log, and the vocabulary matters when scoping which problem a tool is meant to solve. It is also distinct from a model registry, which governs which versioned artifact is promoted to production — tracking answers what was run and with what result, while a registry answers what is deployed and how to roll back, and a tool that does the first does not thereby do the second.

When to use it

Reach for it when:

• More than a handful of training runs, or more than one person doing them
• You need to reproduce or defend a result later
• Comparing configurations systematically rather than by memory
• Where the environment is captured automatically, since that is the half nobody records by hand

Reach for something else when:

• A single throwaway run nobody will need to rebuild
• Logging metrics while leaving data version and commit unrecorded — that is the half that makes it reproducible
• Relied on for rollback, which is a registry's job rather than a tracker's
• Assuming a result can be rebuilt from memory later, which came to one time in six above

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Experiment Tracking — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path