Picking the Model That Scores Your Headlines

Stratify by the thing your archive is most uneven on. For headlines, that is usually rubric band (strong, middling, weak), then format or channel (newsletter subject, homepage headline, ad headline), then whatever else has skewed counts. If 70 percent of your archive is newsletter subjects, a random 15 percent sample will be mostly newsletter subjects, and the model you pick will be tuned to the format you already understand best.

A workable recipe for an archive in the low thousands:

1. Split each stratum into holdout and working pools at 15 to 20 percent holdout.
2. Cap the holdout at a size you can afford to run three times over and still read by hand. Two hundred to four hundred rows is usually enough to see a meaningful gap.
3. Freeze it. Write the row IDs to a file, commit it, and never tune a prompt against those rows again.
4. Keep a second, smaller slice (50 to 100 rows) as a tiebreak set for when two models land within noise of each other.

The temptation to peek is real, and it silently converts your test set into training data. The LangChain write-up on judge calibration is blunt about the failure mode: prompt iteration alone will not close the gap between a technically working evaluator and a trustworthy one, and the systematic path is collecting human corrections and tracking agreement as a measured quantity over time the LangChain guide to calibrating LLM judges.

Three models is the right number for this decision. One is not a comparison, two gives you no tiebreak, and beyond three you spend more time reading outputs than the marginal information is worth.

Hold the prompt constant. Same rubric text, same output schema, same few-shot examples if you use any, same temperature setting. If you change the prompt per model, you are measuring prompt engineering, not model choice. The Label Studio walkthrough of their Prompts feature describes exactly this discipline: you build and version a prompt, run it across model providers, and compare results side by side on the same project, with each version stored alongside its model, instruction and results so you can attribute differences the Label Studio evaluation tutorial.

Ask each model for structured output: a short rationale per dimension, then a constrained label from your rubric's scale. The GoDaddy engineering write-up on score calibration recommends requiring the judge to produce a detailed rationale before a final score, specifically to counter the tendency of judges to cluster scores near the top of the scale the GoDaddy calibration article. The same piece catalogs the biases you will meet: positional bias if you ever show candidates in order, verbosity bias toward longer headlines and rationales, and self-preference when a model judges text that resembles its own output.

For headline scoring, verbosity bias is the one that bites. Long headlines read as informative to a judge even when they violate your character limits. Add an explicit length rule to the rubric and score it as its own yes or no criterion rather than trusting the model to weigh it.

A minimal run record per row looks like this:

{
  "row_id": "h-00412",
  "model": "candidate-b",
  "rubric": {
    "clarity": {"label": "pass", "rationale": "..."},
    "specificity": {"label": "pass", "rationale": "..."},
    "curiosity": {"label": "fail", "rationale": "..."},
    "length_ok": {"label": "pass", "rationale": "..."}
  },
  "input_tokens": 640,
  "output_tokens": 118,
  "latency_ms": 1840,
  "run_index": 1
}

Run each model three times over the holdout. The repeat runs cost little and buy you the stability signal that single-run comparisons miss.

Agreement per dimension. For each rubric dimension, compute the share of holdout rows where the model's label matches your human label. Report it per dimension, not as an average. If your rubric uses graded scales, report both exact agreement and agreement within one step, because a model that is consistently one band conservative is easy to correct and a model that is randomly two bands off is not.

Cost per thousand rows. Multiply the API price per input and output token by the mean tokens per row you measured on the holdout, then scale to 1,000. Include the rationale tokens, since requiring a rationale roughly doubles output length. If your provider bills per image or per request, use that unit instead and state it.

Median latency per row. Use the median, not the mean; one slow request at the tail will distort the average and tell you nothing about the typical row. Note whether you measured at concurrency 1 or under your production concurrency, because queueing changes the number considerably.

The cost-aware model selection study is worth reading before you assume the biggest model wins. Comparing fine-tuned encoder models against zero- and few-shot LLM prompting across four classification benchmarks, it found the encoder-based models were competitive and often superior on macro F1 while operating at one to two orders of magnitude lower cost and latency the cost-aware model selection paper. Headline scoring against a fixed rubric is closer to structured classification than open-ended generation, so the caching and fine-tuning options are live for you too.

Set thresholds before you look at results. A concrete version:

• Weight your dimensions. If clarity and specificity drive clicks and curiosity drives the click decision less than you think, weight accordingly, for example 40, 35 and 25.
• Set a floor per dimension, not just on the weighted total. A model that scores 91 percent on clarity and 62 percent on specificity fails even if the weighted number looks fine, because the failure is concentrated where you cannot afford it.
• Among models clearing every floor, choose the lowest cost per thousand rows at your actual monthly volume.
• Break ties within about five percentage points of agreement by latency if the pipeline is interactive, by cost if it is batch.

Worked example with round numbers, so check your own against your prices. Suppose your holdout is 300 rows, mean input is 650 tokens and mean output including rationale is 200 tokens. Model A costs $0.002 per row, so $2.00 per thousand. Model B costs $0.0004 per row, so $0.40 per thousand. At 50,000 rows a month that is $100 versus $20, a gap of $80 monthly. If Model A clears your floors and Model B misses specificity by nine points, the $80 buys you a rubric dimension you said mattered. If both clear, take Model B and spend the difference on human review of the spot-check queue.

Two rules keep this honest. First, never choose on the basis of the average alone. Second, treat a model that clears your floors by a single point as a tie, because a hundred rows of holdout cannot resolve a one-point difference.

The research on judge reliability supports a modest expectation. Strong LLM judges reach roughly 80 percent agreement with human evaluators, close to the level humans reach with each other, according to the MT-Bench results cited in the LangChain guide the LangChain guide to calibrating LLM judges, and the Confident AI survey puts the figure around 85 percent while noting that the wrong rubric or scoring method leaves evaluation scores as unstable as the model being tested the Confident AI LLM-as-judge guide. Your human labels are also noisy, so a model that agrees with them 85 percent of the time may be near the ceiling. Before you reject a candidate for low agreement, have two editors independently score 30 of the same rows and measure their agreement with each other. That number is your realistic ceiling.

Self-agreement is the signal most teams skip. Run the chosen model three times on the same row and count how often the label is identical. Aggregate that into a per-dimension number. Rows where the model flips between runs are exactly where your rubric is ambiguous or the model is near a decision boundary, and they are the highest-value rows for human attention.

Build the queue weekly: every row with a flipped label, plus every row the model scored as a pass that your rubric treats as essential and the model's rationale is thin on. Send those to an editor. The corrections become few-shot examples, and the example set becomes the thing you re-measure agreement against next quarter.

This is the data flywheel described in the LangChain material: production traces feed observation, observations inform datasets, datasets power evaluations, and evaluations drive the next round of improvements the LangChain guide to calibrating LLM judges. Confident AI's guide makes the same point about scoring methods, recommending techniques such as chain-of-thought prompting, few-shot examples and reference-based scoring to tighten judge behavior before you trust the numbers the Confident AI LLM-as-judge guide.

When you re-run the comparison next quarter, keep the old holdout and add a fresh slice. A model that improved on the new slice but regressed on the frozen one is a warning sign, and the regression-testing pattern is the right mental model: before and after datasets with known differences, and a check for which method actually detects them the Evidently regression testing tutorial. The OpenAI community thread on prompt regression testing captures the operational version of this worry, where a prompt performing well for months suddenly regresses and breaks downstream tools the OpenAI community discussion.

A model that agrees with you 88 percent of the time is not the same as a model that agrees with your editors on the headlines that matter. Weight the holdout rows by business value before you finalize: homepage headlines, paid ad subjects and top-of-funnel emails probably deserve more weight than a mid-article subhead. Ad headline work has its own hard constraints; Google responsive search ads allow 15 headlines at 30 characters each, with double-width characters counting double the Coinis ad headline guide, while SEO headlines aim for roughly 55 to 65 characters before truncation the CopyPress headline guide. If your archive mixes both, add format as a stratification variable and report agreement per format.

The underlying rubric should stay close to the writing advice your editors already use: simple, succinct and specific, with the headline's job being to get the reader to the next line the Between the Lines headline guide. Where a rubric criterion is subjective, such as curiosity, make it a yes or no checklist item rather than a 1 to 5 rating. The GoDaddy calibration work argues for exactly this, replacing vague rating scales with structured yes or no criteria that localize errors and let you see which specific criterion a model fails the GoDaddy calibration article.

If you need to run the three candidates without wiring three SDKs and three billing accounts, a single endpoint for many models removes an afternoon of setup: text, image, video, audio and embedding models through one API key, OpenAI-compatible and prepaid in US dollars, at Quanta Inference.