Eval Harness from Scratch — Building Your Own Evaluation Stack

The Problem

You finally have a model you want to ship. You run it on a few public benchmarks and screenshot the numbers for the launch deck. A month later a customer reports a regression that the benchmark said was solved. Your CFO asks why the vendor benchmark shows a 6% lead but your internal suite shows a 2% loss. A researcher on your team finds that the test set is in the pre-training data.

Each of these failures has the same root cause: the score is not the model; the score is the model plus the harness that produced it. When the harness is a black box — a vendor script, a remote leaderboard, or a README command you copied once — you cannot separate the signal from the wiring.

The hidden wiring includes:

• Prompt template. Does the template ask for only the answer, or does it invite an explanation? The same model can swing 10 points on exact-match tasks.
• Few-shot count. Zero-shot, two-shot, and five-shot are different tasks. A model with strong in-context learning gains more from five examples than a smaller model does.
• Answer extractor. A regex that grabs the first integer, a regex that grabs the last integer, and a strict equality test are three different metrics wearing the same task name.
• Contamination. If the evaluation text appears in the training corpus, the score measures memorization, not capability.

Until these degrees of freedom are explicit and version-controlled, comparing two scores is comparing two unknowns.

The Concept

A map of existing evaluation stacks

The open ecosystem has converged on similar building blocks, but each stack optimizes for a different user.

All five stacks share the same core pipeline, even when the surface looks different:

flowchart LR
    Registry[Task registry] --> Template[Prompt template]
    Template --> Model[Model call]
    Model --> Extract[Answer extractor]
    Extract --> Metric[Metric]
    Metric --> Aggregate[Aggregate / CI / report]

The common abstraction

A task is not a JSONL file. A task is a contract with four functions:

1. prompt_template(example, k, train_examples) -> str — renders the input.
2. extractor(response) -> str — turns free-form model output into a normalized prediction.
3. metric(prediction, reference) -> float — scores the prediction.
4. aggregator(scores) -> float — produces a task-level number and an uncertainty estimate.

The harness in this lesson materializes that contract as a single Task dataclass:

pythonCopy@dataclass
class Task:
    name: str
    prompt_template: Callable
    few_shot: int
    extractor: Callable
    metric: Callable
    examples: List[dict]
    train_examples: List[dict]

Three metric shapes cover most of what engineers actually need:

• Exact match for arithmetic, code output, and closed-class answers:

• Multiple-choice letter extraction for MMLU-style exams. The extractor tolerates "The answer is B" and bare "b" equally:

• Token F1 for short-answer QA where a partial overlap is more informative than zero-or-one:

where T(\cdot) is the set of normalized alphanumeric tokens.

Why build your own

Public stacks are excellent for standardized reporting; they are not excellent for keeping your internal decisions explicit. When you own the harness:

• You can vary one freedom at a time and measure the score delta. That delta is often larger than the model delta you are trying to evaluate.
• You can add a contamination check before you trust a number.
• You can plug in a real model by replacing MockModel.generate with an API call. The rest of the harness does not change.

Build It

The runnable artifact is code/main.py. It is intentionally small — everything fits in one file — but it is not a toy. It is the shape a production harness should have.

Run it:

consoleCopypython3 main.py

You will see three things:

1. A per-task score table with 95% bootstrap confidence intervals and a macro average.
2. A comparability experiment that runs the same seeded MockModel through five arithmetic-harness variants and prints the score drift.
3. An 8-gram contamination check on a clean query and a leaked query.

The MockModel

MockModel is the seam. It takes correctness and format_ok probabilities and a seed, so the harness is testable without an API key:

pythonCopymodel = MockModel(correctness=0.8, format_ok=0.75, seed=42, name="mock-80/75")

With probability 1 - format_ok the model returns a noisy wrapper like "I think the answer is 33". The harness's extractor either recovers the answer or fails, which is exactly what happens with real models that do not follow instructions.

The runner

pythonCopydef run_task(task: Task, model) -> Dict:
    scores = []
    for ex in task.examples:
        prompt = task.prompt_template(ex, task.few_shot, task.train_examples)
        raw = model.generate(ex, task.name)
        pred = task.extractor(raw)
        scores.append(task.metric(pred, ex["answer"]))
    return {"scores": scores, "mean": float(np.mean(scores))}

run_leaderboard loops over tasks, adds a bootstrap CI, and reports the macro average. The CI is computed by resampling the per-example scores with replacement:

pythonCopydef bootstrap_ci(scores, n_boot=2000, seed=0):
    arr = np.asarray(scores, dtype=float)
    rng = np.random.default_rng(seed)
    means = np.array([
        arr[rng.integers(0, arr.size, size=arr.size)].mean()
        for _ in range(n_boot)
    ])
    return float(np.percentile(means, 2.5)), float(np.percentile(means, 97.5))

The comparability experiment

The demo runs the same MockModel through five arithmetic configurations:

textCopybaseline (format instruction, k=2, strict)  -> 0.800
few-shot=0                                  -> 0.500
chatty template                             -> 0.500
plain template                              -> 0.200
lenient extractor                           -> 0.900

The numbers will vary slightly with the seed, but the pattern is robust: the reported score is a function of the harness, not just the model. That is the lesson. A team that does not version-control these choices is comparing weather reports from different cities.

Contamination detection

Before you trust a benchmark, check whether its text appears in your training corpus. The detector builds word 8-grams from the evaluation example and intersects them with 8-grams from each corpus document:

pythonCopydef contamination_check(test_text, corpus, n=8):
    test_grams = _ngrams(test_text, n)
    hits = []
    for doc_id, doc in enumerate(corpus):
        overlap = test_grams & _ngrams(doc, n)
        if overlap:
            hits.append({"doc_id": doc_id, "ngrams": sorted(overlap)})
    return hits

A single long shared sentence is enough to flag the example. The harness does not throw the score away; it labels it so the report can disclose the overlap.

Verify It

code/tests/test_main.py pins the parts that make this a harness rather than a script:

• letter_extractor recovers a choice letter from a full sentence and from lowercase input.
• lenient_number_extractor recovers the last integer from a noisy response.
• exact_match is invariant to case and whitespace but not to semantic equivalence.
• token_f1 gives partial credit on overlapping short answers and zero on disjoint answers.
• bootstrap_ci contains the true mean of a symmetric score vector.
• MockModel with format_ok=1.0 returns the raw answer; with format_ok=0.0 it returns a noisy wrapper.
• Running the same model with a strict extractor versus a lenient extractor produces a different mean score.
• contamination_check flags a leaked 8-gram and ignores a clean query.

Why This Matters for Engineers

• Regression testing needs an owned harness. Before you ship a new checkpoint, rerun the same task registry, the same templates, and the same extractors. Any score change is the model, not the measurement.
• Vendor selection needs a common harness. When two providers quote different benchmark numbers, ask for the exact prompt, few-shot count, and extractor. The delta often disappears once the harness is fixed.
• Executive reporting needs uncertainty. A single mean without a confidence interval or contamination check is a number without context. Bootstrap CIs and overlap reports turn a score into a defensible claim.
• Benchmark overfitting is real. If your team optimizes against a leaderboard you do not control, the only defense is a private harness with tasks the leaderboard cannot see. This lesson ships the skeleton of that harness.

Hints

• Keep the Task dataclass small. If a task needs a special extractor, write a function and pass it in; do not add fields to the core class.
• Seed the MockModel before every test that asserts on score ordering. Stochastic coverage is fine for exploration, deterministic seeds are required for assertions.
• When you add a real model, replace only MockModel.generate. The runner, metrics, and aggregators stay the same.
• If you add more tasks, prefer per-example confidence intervals over global ones. The macro average can hide a task that collapsed to zero.
• Version-control the prompt templates as strings in code, not as ad-hoc strings in notebooks. A one-line template change is a one-line diff in git.

Key Terms

• Task registry: the centralized list of evaluation tasks, each bound to its template, extractor, and metric.
• Answer extractor: the function that converts a model's free-form output into a normalized prediction.
• Macro average: the unweighted mean of per-task scores; treats every task as equally important.
• Bootstrap CI: a confidence interval built by resampling the observed scores with replacement.
• n-gram contamination: overlap between evaluation text and training text above a chosen n-gram length, used as a memorization warning.
• Score comparability: the property that two reported scores were produced by the same harness choices, so their difference is attributable to the model.