Test-Driven Prompt Engineering with DSPy

Article

This is an advanced take on prompt engineering. If you are newer to the basics, start there first.

What is test-driven prompt engineering?

Test-driven prompt engineering is a way of improving prompts by measuring them, not by feel. You write a metric that defines success, then change the prompt only when the metric goes up.

The idea borrows directly from software testing. In code, you do not merge a change because it looks right, you merge it because the tests pass. TDPE applies the same rule to prompts.

The mindset shift is small but powerful. A prompt without a test is a guess, and guesses do not survive contact with real inputs or a new model.

Why hand-tuning does not scale

Hand-tuning works for a demo and breaks in production. Three problems show up fast.

There is no metric, so you cannot tell whether "think step by step" actually beats "reason carefully" without running both against data. Prompts are fragile, so wording that shines on one model can degrade on another. And systems grow, so a pipeline with a dozen prompt-driven steps becomes impossible to tune by eye.

The internet is full of "top 50 prompts" lists that ignore all three. They optimize for looking clever, not for a number you can defend.

The TDPE loop

Test-driven prompt engineering runs a short, repeatable loop. It mirrors the write-test-refactor cycle you already know.

1. Specify. State the task precisely: inputs, outputs, and what "correct" means. 2. Build an eval set. Collect real examples with known-good answers. 3. Define a metric. Turn "correct" into a function that returns a score. 4. Baseline. Measure the current prompt so you have something to beat. 5. Optimize. Improve the prompt against the metric. 6. Regression-test. Lock the score in as a test so future changes cannot quietly break it.

The loop is the whole discipline. Everything DSPy does is automate steps four through six.

What is DSPy?

DSPy is a framework from Stanford NLP for programming language models instead of prompting them. Rather than writing a prompt string, you declare what a step should do and let DSPy generate and optimize the actual prompt.

It has three core pieces. Signatures declare a task's typed inputs and outputs. Modules wrap a signature in a strategy such as chain-of-thought. Optimizers compile your program by searching for better instructions and examples against your metric.

The result is a prompt you did not hand-write, tuned to a number you chose. The official DSPy project is the reference for the current API.

Signatures and modules

A signature describes the task, not the prompt. DSPy turns the type hints and docstring into the underlying instruction.

```python import dspy

dspy.configure(lm=dspy.LM("openai/gpt-5.4")) # any provider works

Declare the task as a typed signature, not a prompt string. class Classify(dspy.Signature): """Label a support ticket as billing, technical, or account.""" ticket: str = dspy.InputField() label: str = dspy.OutputField(desc="one of: billing, technical, account")

Wrap it in a reasoning module. program = dspy.ChainOfThought(Classify) ```

You never wrote "You are a helpful classifier." DSPy builds that for you, and the optimizer will improve it. Modules such as dspy.ReAct extend the same idea to tool-calling agents.

Writing the eval as a test

The metric is where TDPE lives. It is a plain function that scores a prediction, and it doubles as your test's assertion.

```python The metric: your definition of "correct". def accuracy(example, pred, trace=None): return example.label.lower() == pred.label.lower()

The regression test: fail the build if quality drops. def test_classifier_meets_bar(): evaluate = dspy.Evaluate(devset=testset, metric=accuracy) score = evaluate(program) assert score >= 0.90, f"Prompt regression: {score:.2f} < 0.90" ```

Now a prompt change that lowers accuracy fails CI, exactly like a broken unit test. That single guardrail is what turns prompting from folklore into engineering.

Optimizers: compile your prompt

An optimizer searches for a better prompt against your metric. You author the task, define the metric, and let DSPy do the tuning in-process.

```python Compile: DSPy searches instructions and few-shot demos against your data. optimizer = dspy.MIPROv2(metric=accuracy, auto="light") optimized = optimizer.compile(program, trainset=trainset) ```

DSPy ships several optimizers, and picking one is mostly about budget and data size.

| Optimizer | What it does | Reach for it when | |---|---|---| | BootstrapFewShot | Generates good few-shot examples | Getting started, small datasets | | MIPROv2 | Jointly tunes instructions and demos with Bayesian search | The workhorse default | | GEPA | Reflects on run traces and textual feedback to rewrite prompts | Complex reasoning, when you can give feedback | | SIMBA | Mini-batch sampling plus introspective failure analysis | Larger datasets |

Start with the cheapest optimizer that clears your eval, and only climb the ladder when it does not.

The model-swap problem

Here is the failure hand-tuning hides. A prompt carefully tuned for one model often degrades on another, because each model responds to instructions differently.

TDPE fixes this cleanly. Because the task is defined by a signature and a metric, not a frozen string, you re-compile for the new model and measure the result.

```python Same program, re-compiled for a different model. dspy.configure(lm=dspy.LM("anthropic/claude-sonnet-5")) optimized_v2 = optimizer.compile(program, trainset=trainset) ```

Your prompt is no longer married to one model. It is a program you can retarget, which matters every time you compare or switch providers.

Boundary and safety tests

Metrics are not only for accuracy. You can test the behavior you never want, the same way you test the behavior you do.

Write metrics that check refusals, output format, and alignment, then assert on them. A test can confirm the model declines an out-of-scope request or never returns unescaped output. This is the prompt-side complement to defending against prompt injection: you encode the boundary as a test and fail the build when it breaks.

Treat safety as a first-class metric, not an afterthought you check by hand.

When not to use DSPy

TDPE is powerful, but it is not free, and small tasks do not need it.

Skip the framework when a prompt is a one-off, when you have no evaluation data to optimize against, or when the task is so small that a metric would take longer to write than the prompt. DSPy earns its keep on prompts you will maintain, reuse, or run at scale.

The deciding question is simple. If you cannot write a metric for the task, you are not ready to optimize it, with or without DSPy.

Sources

Stanford NLP, DSPy (official repository and documentation). Signatures, modules, the optimizer lineup (BootstrapFewShot, MIPROv2, GEPA, SIMBA), and the compile-against-a-metric workflow.

Methodology: the TDPE concept is asserted in ModelRefs' own voice as the reference layer, and all DSPy API details are cross-checked against the official DSPy project on 17 Jul 2026. Code is illustrative; version-sensitive syntax should be re-verified before publication.