How to Evaluate Prompts Programmatically

Article

This is the from-scratch companion to our guide on test-driven prompt engineering with DSPy. Build the harness here to understand the mechanics, then reach for DSPy when you want the optimization automated.

Why programmatic evaluation

Manual checking does not scale and does not catch regressions. A prompt that looks fine on three examples can fail on the fourth, and a change that helps one case can quietly break another.

A code-based harness fixes both. You measure every change against the same dataset, so improvement is a number, not an impression. This is the same idea behind OpenAI Evals and every serious eval stack: an eval is just a dataset plus a metric, run on every change.

The payoff compounds. Once the harness exists, every production failure becomes a new test case, and your coverage grows toward the problems you actually hit.

What you'll build

A reusable harness with four moving parts. The dataset holds cases, the metric scores one output, the runner scores a whole prompt, and the report tells you what failed.

The flow is: a dataset of cases with expected answers and a prompt feed into a runner, which calls the model and scores each output with a metric. The runner produces a report with the overall score and a list of failures. If the score meets the bar, you ship; if not, you fix the prompt and re-run.

Prerequisites

- Python 3.10+. The core harness needs nothing else. - A provider SDK (openai or anthropic) only for Step 7, when you swap the stub for a real model.

Steps 1 through 6 run with no API key and no network, so the output is fully reproducible.

Step 1. Define your dataset

An eval starts with examples that have known-good answers. Keep them in code so they version with everything else.

```python eval_harness.py from dataclasses import dataclass

@dataclass class Case: text: str expected: str

A support-ticket routing task. Grow this from real tickets over time. DATASET = [ Case("I was charged twice this month", "billing"), Case("The app crashes when I upload a file", "technical"), Case("I can't reset my password", "account"), Case("My invoice total looks wrong", "billing"), Case("The API returns a 500 error", "technical"), Case("I want to close my account and get a refund", "account"), ] ```

The last case is deliberately tricky. It mentions both an account action and a refund, and it will expose a weakness in a moment.

Step 2. Define the model under test

The harness calls a model through one function. For a reproducible tutorial we start with a deterministic stub, then swap in a real provider in Step 7.

```python def call_model(prompt: str, text: str) -> str: """Deterministic stub so this tutorial is reproducible. A placeholder for a real model; it ignores the prompt and matches keywords.""" t = text.lower() if any(w in t for w in ("charge", "invoice", "refund", "bill")): return "billing" if any(w in t for w in ("crash", "error", "500", "bug", "upload")): return "technical" if any(w in t for w in ("password", "login", "account", "reset")): return "account" return "unknown" ```

Routing every call through one function is the key design choice. It lets you swap models without touching the rest of the harness.

Step 3. Write a metric

A metric turns "correct" into a function. Start with exact match, which fits classification and extraction.

```python def exact_match(expected: str, actual: str) -> bool: return expected.strip().lower() == actual.strip().lower() ```

Exact match is strict and cheap. For open-ended text you will want a softer metric, covered in the metric table below.

Step 4. Build the runner

The runner applies the model to every case, scores each with the metric, and collects the results into a report.

```python @dataclass class Result: case: Case actual: str passed: bool

class Report: def __init__(self, results: list): self.results = results

@property def score(self) -> float: return sum(r.passed for r in self.results) / len(self.results)

@property def failures(self) -> list: return [r for r in self.results if not r.passed]

def evaluate(prompt: str, dataset: list, metric=exact_match, model=call_model) -> Report: results = [] for case in dataset: actual = model(prompt, case.text) results.append(Result(case, actual, metric(case.expected, actual))) return Report(results) ```

That is the whole engine. Everything else is a dataset, a metric, or a way to read the report.

Step 5. Run it and read the report

Now run the harness and print the score plus any failures.

```python PROMPT_V1 = "Classify the ticket as billing, technical, or account."

report = evaluate(PROMPT_V1, DATASET) print(f"Score: {report.score:.0%} ({len(report.failures)} failed)") for r in report.failures: print(f" FAIL: {r.case.text!r} -> got {r.actual!r}, expected {r.case.expected!r}") ```

Paste blocks 1 through 5 into eval_harness.py and run python eval_harness.py. You get exactly this:

```text Score: 83% (1 failed) FAIL: 'I want to close my account and get a refund' -> got 'billing', expected 'account' ```

The report earns its keep immediately. It caught the ambiguous case, where "refund" pulled the answer to billing instead of account. That is a real failure you would have missed by eyeballing.

Step 6. Add a regression gate

Wrap the score in an assertion and it becomes a test. Drop this into your suite and a bad prompt change fails CI.

```python def test_classifier_meets_bar(): report = evaluate(PROMPT_V1, DATASET) assert report.score >= 0.80, f"Regression: {report.score:.0%} < 80%" ```

Run it with pytest and it passes at 83 percent. Lower the bar and you lose the guardrail; raise it above your current score and the test tells you there is work to do. Either way, quality is now enforced, not hoped for.

Step 7. Swap in a real model

The stub proved the mechanics. To evaluate a real prompt, replace call_model with a provider call. The rest of the harness does not change.

```python from openai import OpenAI client = OpenAI()

def call_model(prompt: str, text: str) -> str: resp = client.chat.completions.create( model="gpt-5.4", messages=[ {"role": "system", "content": prompt}, {"role": "user", "content": text}, ], ) return resp.choices[0].message.content.strip() ```

Real model output varies between runs, so exact scores will differ from the stub. That is expected. The harness, the metric, and the gate stay identical, which is the point.

Step 8. Compare prompts or models

Because the runner is reusable, comparison is just running it twice. This is how you settle "is v2 actually better" with evidence.

```python PROMPT_V2 = ( "Classify the support ticket into exactly one of: billing, technical, account. " "If the ticket is about closing or managing an account, prefer 'account'." )

def compare(prompts: dict, dataset: list) -> None: for name, prompt in prompts.items(): r = evaluate(prompt, dataset) print(f"{name:4} {r.score:.0%}")

compare({"v1": PROMPT_V1, "v2": PROMPT_V2}, DATASET) ```

Run this against a real model and you get a side-by-side score. Swap the loop to hold the prompt fixed and vary the model, and the same harness becomes a model bake-off, useful whenever you are choosing between providers.

Choosing a metric

Exact match is the start, not the whole story. Match the metric to the task.

| Metric | Best for | Trade-off | |---|---|---| | Exact match | Classification, labels, structured fields | Too strict for free text | | Contains / keyword | Checking a required fact appears | Misses meaning, easy to game | | Structured-schema check | JSON or tool-call outputs | Validates shape, not correctness | | Semantic similarity | Paraphrase-tolerant answers | Needs an embedding model; fuzzy threshold | | LLM-as-judge | Open-ended quality, tone, helpfulness | Judge bias; use a different model family, keep a rubric |

For anything high-stakes, keep a strict check as ground truth and use an LLM judge only where exact scoring is impossible. The same discipline appears in our guide to testing an AI agent safely.

Common mistakes

Most broken evals share a few roots.

Teams test on too few cases, so the score is noise. They grade free text with exact match, so good answers fail on wording. They forget to add each production failure back to the dataset, so the suite never learns. And they trust an LLM judge blindly, without a rubric or a human spot-check.

Fix these by growing the dataset from real traffic, matching the metric to the task, and treating the judge as a signal rather than a verdict.

Sources

- OpenAI, Evals (open-source framework) -- prior art for code-based evaluation: an eval is a dataset plus a metric, re-run on every prompt or model change to catch regressions.

Methodology: the harness and its patterns are asserted in ModelRefs' own voice as the reference layer. Steps 1 through 6 are deterministic and reproduce the output shown; real-model steps are labeled as varying. Cross-checked against OpenAI Evals on 17 Jul 2026.