ModelRefs / Hyperparameter — AI Glossary

Hyperparameter — AI Glossary

A configuration value set before training (learning rate, batch size, LoRA rank) that governs the training process but is not learned from data.

Overview

Hyperparameters control how a model learns, contrasted with parameters (weights) which are learned. Key LLM hyperparameters: learning rate, warmup steps, weight decay, batch size, context length, LoRA rank/alpha, and dropout rate. Hyperparameter tuning methods: grid search, random search, Bayesian optimization (W&B Sweeps, Optuna). Poor hyperparameter choices can prevent convergence or cause overfitting.

Reference details

Topic	training
Last reviewed	2026-06-24

How it works

Hyperparameters are fixed before the optimiser runs and are not touched by gradients, which is precisely what separates them from the weights.
Each one enters the training dynamics at a different point — the learning rate scales every update, batch size sets gradient variance, weight decay adds a pull toward zero.
Their effects interact rather than add, so the best learning rate depends on the batch size and the schedule, which is why single-parameter sweeps mislead.
A search strategy proposes configurations, each evaluated by training to some checkpoint and reading a validation metric.
Grid search enumerates the cross-product, random search samples it, and Bayesian methods fit a surrogate over observed results to propose the next point.
The practically important asymmetry is that only a few hyperparameters usually matter much, and which few is not known before the search begins.

Example: Why random search beats grid search on the same budget

Take 5 hyperparameters at 4 candidate values each. Grid search enumerates 4^5 = 1,024 configurations, and at 40 minutes per training run that is 40,960 minutes, which is 40,960 / 60 ≈ 682.7 hours or about 28.4 days of sequential compute. Now suppose, as is typical, that only 2 of the 5 genuinely affect the outcome. Random search samples configurations independently, so each trial is an independent draw in the 2-dimensional subspace that matters, and the chance that n trials all miss the top 5% of that subspace is 0.95^n. For n = 60 that is 0.95^60 ≈ 0.0461, so the probability of landing in the top 5% is about 95.4%. Sixty runs at 40 minutes is 2,400 minutes, exactly 40 hours. So the comparison is 1,024 runs over about 682.7 hours against 60 runs over 40 hours for a 95.4% chance of a near-best configuration — a 1,024 / 60 ≈ 17.1-fold reduction in runs. The reason is structural rather than statistical luck: the grid spends its budget on resolution along axes that do not matter, testing 4 distinct values of three irrelevant parameters, while random search gives every trial a fresh position in the subspace that does. Grid search only wins when every parameter matters and the grid's resolution is the quantity of interest.

Commonly confused with

A hyperparameter is not a parameter, and the shared root causes genuine confusion in reading papers and configs. Parameters are the weights, learned from data by gradient descent, and a 7B model has about seven billion of them. Hyperparameters are set by a human or a search before training, number in the tens, and govern how the learning happens. The second confusion is with inference-time settings — temperature, top-p, repetition penalty — which are frequently called hyperparameters and are not: they alter how a finished model is sampled, change nothing about the weights, and can be adjusted per request. The distinction matters operationally, because a bad learning rate requires retraining while a bad temperature requires a different API argument.

When to use it

Reach for it when:

When a training run underperforms and the architecture and data are not in question, where the learning rate is the usual first suspect.
With random rather than grid search on a fixed budget, as 60 trials gave a 95.4% chance of a top-5% configuration above.
For PEFT adaptation, where rank and alpha interact with the learning rate and defaults transfer poorly across tasks.
With Bayesian optimisation once the search space is narrowed, where a surrogate over observed results outperforms sampling.

Reach for something else when:

As an explanation for a model that cannot do the task, which is a capacity or data problem rather than a tuning one.
Via grid search over many dimensions, which spent about 682.7 hours on resolution along axes that may not matter.
Conflated with decoding parameters, which are per-request settings on a finished model.
Tuned on the test set, which converts a measurement of generalisation into a measurement of the search.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Hyperparameter — AI Glossary.

Frequently asked questions

What is Hyperparameter?