Active Prompting — AI Glossary

Reference details

Related terms

• Few-Shot Prompting
• Chain of Thought (CoT)
• Calibration

How it works

1. A pool of unlabelled questions is assembled from the target task, and the method's goal is to decide which few of them are worth paying a human to annotate with reasoning.
2. The model answers each pool question several times with sampling enabled, producing a distribution of answers rather than a single one.
3. Uncertainty is computed from that distribution — disagreement among the samples, or the entropy of the answer frequencies. High disagreement marks a question the model finds genuinely hard.
4. The most uncertain questions are selected for human annotation, on the reasoning that an example teaches most where the model is least settled.
5. Humans write chain-of-thought reasoning for those selected questions, and those become the few-shot exemplars in the prompt.
6. The loop can repeat: with new exemplars in place, uncertainty is recomputed and the next batch is selected against the improved baseline.

Example: What the selection buys against the sampling it costs

Suppose an annotation budget of 8 examples and a pool of 500 questions. Random selection draws 8 from the pool, so the chance that any given question is picked is 8 / 500 = 1.6%, and the expected number of genuinely hard questions among the 8 is 8 times the pool's hard fraction — if 20% are hard, that is 8 x 0.2 = 1.6. Active selection by uncertainty targets them directly, so close to all 8 can be hard: roughly a 5x improvement in how much of the budget lands where the model needs help. The cost is the uncertainty estimation itself: 500 questions at 10 samples each is 5,000 model calls before a single annotation is written. Whether that trades well depends entirely on the relative prices. If human annotation of one chain-of-thought example costs the equivalent of 2,000 model calls, the 5,000 calls are worth 2.5 annotations — so spending them to make 8 annotations roughly 5x better is clearly positive. If annotation is cheap and the model is expensive, the arithmetic reverses and random selection with a larger budget wins.

Commonly confused with

Active prompting selects which examples to annotate; it does not select which annotated examples to include in a prompt at query time. The second thing — retrieving the nearest labelled examples for each incoming question — is dynamic exemplar selection, and the two are complementary: active prompting builds the labelled pool efficiently, dynamic selection uses it per request. It is also not active learning in the classical sense, which selects data for fine-tuning and updates weights; here nothing is trained and the output is a prompt.

When to use it

Reach for it when:

• A fixed annotation budget and a pool far larger than it, which is the situation the method is designed for.
• Reasoning tasks where chain-of-thought exemplars measurably help, since that is what is being annotated.
• Expensive annotation — expert domains where each labelled example costs real money or specialist time.
• Stable task distributions, so exemplars selected once keep earning against future questions.

Reach for something else when:

• Cheap or plentiful annotation, where the sampling cost of estimating uncertainty exceeds what better selection saves.
• Tasks where few-shot examples do not help, in which case the annotation budget is misdirected regardless of selection.
• Small pools, where selection has little to choose between and random is nearly as good.
• Shifting distributions, where exemplars chosen against yesterday's uncertainty do not match today's questions.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Active Prompting — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path