How to Run a Model Bake-Off: Script, Score and Sign-Off

The single biggest determinant of a useful bake-off is the dataset. A practical evaluation framework treats the dataset as having a defined scope that matches the task, inputs that mirror production usage, and enough variety to avoid a narrow test the practical evaluation guide on arXiv. That guide also insists the data be kept separate from anything used in training and treated as a living set that changes as your application changes.

Here is a sampling procedure that takes about a day and works for teams without a research function.

1. Export a week or a month of real requests: support tickets, drafting requests, extraction jobs, classification calls. Whatever the model will actually do.
2. Stratify by task type and difficulty. If 40 percent of tickets are refund requests, the eval set should be roughly 40 percent refund requests.
3. Pull out 30 to 60 items. Below about 30, single items swing the average; above about 60, reviewer time explodes before the evidence gets better.
4. Strip personally identifying information and anything regulated. Keep the shape of the input, not the person.
5. Write the expected output or the rubric that defines a good answer. For open-ended tasks, the rubric is the expected output.
6. Freeze the set in version control before any model runs. Changes after the fact must be versioned, not quietly edited.

For extraction, classification and similar closed tasks, you can compare model output to a known ground truth mechanically. For drafting and reasoning tasks, you cannot, and you will need structured human or model-assisted judgment the Evidently AI guide to LLM evaluation. Know which regime you are in before you build the harness, because it changes the scoring sheet.

A bake-off compares candidates. If prompts differ, you compared prompt engineering, not models. Put every item through a single runner with a config per candidate.

import json, time

def run_candidate(client, model_cfg, items, out_path):
    rows = []
    for item in items:
        start = time.perf_counter()
        resp = client.completions.create(
            model=model_cfg["id"],
            prompt=item["prompt"],
            temperature=model_cfg["temperature"],
            max_tokens=model_cfg["max_tokens"],
            stop=model_cfg.get("stop"),
        )
        elapsed = time.perf_counter() - start
        rows.append({
            "item_id": item["id"],
            "candidate": model_cfg["name"],
            "output": resp.text,
            "latency_s": round(elapsed, 2),
            "tokens_out": resp.usage.output_tokens,
        })
    with open(out_path, "w") as f:
        json.dump(rows, f, indent=2)

Three things this snippet deliberately fixes. Temperature and max tokens are pinned per candidate so length and randomness do not drift. Latency is measured in the same script, in the same region, because inference latency depends on request queuing, prefill and network conditions, and long prompts push time to first token upward NVIDIA's benchmarking primer. And output tokens are logged, because cost per task is a function of tokens, not of vibes.

If candidates expose different parameter names or defaults, record the mapping in a config file and keep it in the same folder as the outputs. Anyone reviewing the decision six months later needs to reconstruct what ran.

Blind review is where most bake-offs gain or lose credibility. Rename outputs to opaque identifiers before review: cand_A, cand_B, cand_C, with the mapping held by one person who is not scoring. Reviewers see the task input, the rubric and the anonymized outputs in random order.

Rubric design deserves more care than it usually gets. Rubric-guided evaluation decomposes vague quality into atomic, checkable constraints, and the approach is spreading precisely because free-form judgments track presentation over substance RubricBench on arXiv. That paper also finds a substantial gap between human-annotated rubrics and model-generated ones, which is a useful warning: write the rubric with a domain expert, not by asking a chatbot to draft it.

A concrete rubric for a customer email reply might look like this.

Practical mechanics that prevent quiet corruption of the results:

• Two reviewers per output minimum. Single reviewers drift, and drift is invisible without a second score.
• Record disagreements verbatim. "Reviewer 1 scored accuracy 5, reviewer 2 scored 2" is data. Resolve it in a discussion and log the resolution.
• Randomize order per reviewer. Whitelabel models sometimes reveal themselves in formatting quirks, and reviewers remember the third output they saw.
• Score dimensions, not overall impressions. Overall scores collapse to "which one felt better."

Model-assisted judges can scale this, and rubrics make the judge's job feasible, but automated judging carries known biases including verbosity bias and self-preference, so it needs calibration against human scores the Medium analysis of rubric-based judging. For a bake-off of 30 to 60 items, human review is usually cheaper than building a trustworthy judge, and the reviewers learn something your stakeholders will ask about anyway.

Here is a worked example with three candidates, five criteria and weights that sum to 100 percent.

The arithmetic for Candidate A is 0.30×4.0 + 0.15×3.5 + 0.10×4.5 + 0.15×3.0 + 0.15×4.0 + 0.10×3.0 + 0.05×4.0 = 3.73. Multiply each score by its weight and sum, exactly as the weighted scoring walkthroughs describe the GoLeanSixSigma template guide and ProductPlan's definition.

Now the insight the spreadsheet does not give you. B wins by two hundredths of a point over C, which is inside the noise of a 30-item eval set with two reviewers per item. The decision rule that follows: treat candidates within roughly 5 percent of the top score as a tie on quality, then break ties with operational criteria your matrix weights lightly, such as latency variance, vendor concentration risk or the existence of a working integration path. Here C wins the tie break for most teams because latency and cost scores are measured with far less error than tone scores. That is a defensible decision, and the spreadsheet still explains it.

One more number worth computing: sensitivity. Re-run the totals with accuracy weighted at 40 percent instead of 30 percent and see if the ranking flips. If it does, your decision hinges entirely on how much you care about accuracy, and you should say so explicitly in the memo rather than pretending the total settled it. It is also worth noting that many organizations are still early in measuring AI outcomes at all; transaction data shows newer firms reaching a 10 percent adoption rate in six months while the earliest cohort took over six years, and employer firms adopted at nearly twice the rate of nonemployers the JPMorgan Chase Institute report. A documented bake-off is often the first piece of evidence a smaller organization has.

The memo is short. One to three pages, with the scorecard as an appendix. Structure it like this.

1. Decision requested. One sentence: which candidate to adopt, for which use case, effective when.
2. Criteria and weights. Pasted from the pre-registered rubric, with the date they were fixed.
3. Eval set description. Item count, task mix, source of the items, version tag.
4. Results table. The weighted totals plus the underlying dimension scores and reviewer counts.
5. Disagreements and how they were resolved. Name the specific items where reviewers split, and what the resolution was.
6. Tie-break rationale. For candidates within 5 percent, why the chosen one won on operational grounds.
7. Risks and exit plan. What would make you reverse the decision, and what you will measure to find out.
8. Sign-off block. Name, role, date, and what specifically each signer is approving.

Point 3 matters more than teams expect. A decision matrix's outputs are only as good as the data behind them, and keeping the data reliable and current is listed as a core limitation of the method itself the Lucid explainer. When someone challenges the decision three months later, the eval set version is the answer.

When the bake-off is done and the winning model is chosen for a production path, routing several model types through one interface removes a chunk of integration work from the next comparison. Quanta Inference exposes text, image, video, audio and embedding models behind one OpenAI-compatible, prepaid API key priced in US dollars, which makes swapping the chosen candidate in and re-running the same harness a configuration change rather than a rewrite.

Most bake-off problems are visible before the scoring starts if you check for them.

The length bias row is worth emphasizing because it is the most common way an unwitting bake-off produces a worse outcome. Evaluation methods based on judging reward verbosity and surface complexity over actual problem resolution RubricBench on arXiv, and the same bias shows up in human reviewers who skim. Put the length constraint in the rubric as a scored dimension, not as an instruction in the prompt that the model may ignore.

Finally, define done before you start. A bake-off is finished when the memo is signed, the outcomes are logged against a measurable baseline, and a date exists for re-running the set. If you would rather have the evaluation designed for you and the outputs scored by someone who has run these before, a free 30-minute discovery call is the low-commitment version of that conversation.