Building a Headline Regression Set from Your Own Archive

The first job is joining two tables you probably already have: the headline text and the result it produced.

Export from your content system at the story level, one row per headline, with the original text exactly as published, the publish date, the channel (site, newsletter, social), the placement, and the outcome metric you care about. Keep the raw headline string untouched, including casing and punctuation, because character-level details change how a headline reads. If your page includes the destination URL, a short description and the headline, keep all three so the scorer judges the headline in context.

Selection rules that save you months of pain:

• One metric per set. Pick click-through rate, engaged reads or conversion, not all three. If you mix them, a headline that wins on clicks and loses on reads will get two conflicting labels.
• Fixed windows. Measure each headline over the same post-publish window (for example the first 48 hours) so a slow-burn evergreen piece is not compared against a spike piece.
• Comparable placement. A homepage hero and a buried newsletter item are different jobs. Stratify by placement and sample from each stratum.
• Exclude the confounded. Paid boosts, A/B tests with split traffic, syndicated copies and anything published during an outage go in a reject list, not the set.

A few hundred rows is enough to start. If you have 60% winners and 40% losers across 300 headlines, you have 180 and 120 in each class, plenty to estimate agreement with reasonable error.

This is where most regression sets quietly fail. Two people read "10 Signs You Need a New Roof" and disagree about whether it "creates curiosity." Write the rubric down, with yes or no criteria, and apply it before you ever involve a model.

The GoDaddy engineering write-up on calibrating judge scores lays out the standard biases and their fixes: positional bias (randomize order), verbosity bias (add a conciseness criterion), overly positive skew (force a rationale before the score), and prompt sensitivity (normalize against a small human-labeled gold set) (GoDaddy, Calibrating Scores of LLM-as-a-Judge). Their recommendation of rubrics with verifiable yes or no criteria maps directly onto headline labeling.

A workable label scheme:

Write down three example headlines per level and pin them to the rubric document. Then have two people label 30 overlapping headlines blind and measure agreement. If they disagree on more than a handful, the rubric is not specific enough; fix the wording before scaling up.

The most important decision: label from the rubric, not from the outcome. Ask "does this headline meet the criteria for a 3?" and keep the measured outcome in a separate column. If you label from the metric, the set only teaches the judge to copy historical results, and a genuinely better headline that underperformed for reasons outside the copy will get punished forever. Keep both columns and use the outcome only as a sanity check on the rubric, for example to flag cases where a rubric-3 headline had a median or worse click rate.

That checklist discipline is the same principle used in structured evaluation forms elsewhere: a row of verifiable criteria beats a vague overall impression, and it lets you pinpoint where a judgment went wrong (an example rubric structure).

The set is a test asset, so treat it like code. One file per version, one row per headline, and a changelog entry whenever you add or relabel a row.

{
  "set_version": "hl-2024-03",
  "rubric_version": "rubric-v3",
  "rows": [
    {
      "id": "hl-00412",
      "headline": "How to Cut Your Water Bill by Half",
      "channel": "newsletter",
      "placement": "top_story",
      "window_hours": 48,
      "outcome_ctr": 0.041,
      "label": 3,
      "labeled_by": ["reviewer_a", "reviewer_b"],
      "notes": "both raters agreed"
    }
  ]
}

Store the original headline verbatim, never a cleaned or re-cased copy. Freeze the rubric text alongside the set so that a future reader can tell whether a label was made under old or new criteria. And record the rater IDs, because when agreement later drops you will want to know whether the labels or the judge changed.

Running the set is a three-step loop: score every row with the current prompt and model, map the model's score onto your label scale, then compare.

The first run is the baseline. Save the scores, the model identity, the prompt hash and the date. Nothing else happens until you have that baseline. Do not tune the prompt while establishing it or you will never know what "normal" looks like.

For each subsequent run, compute two numbers per headline:

• The judge score, mapped to the same 1 to 3 scale as your label.
• The absolute difference between judge score and label.

Then aggregate. Report exact match rate, "within one level" rate, and the count of severe misses (judge says 3, label says 1). The stringent-agreement framing used in calibrated rubric work, where predictions for each judge are compared against actual human ratings rather than a single average, is a useful model here (Microsoft's LLM-Rubric paper). You do not need their calibration network; you need the habit of measuring agreement per item and then aggregating.

A worked example with numbers. Suppose your baseline run on 300 headlines gives 71% exact match and 94% within one level, with 5 severe misses. You rewrite the prompt to reward specificity and rerun. Exact match falls to 62%, but within-one-level rises to 96% and severe misses drop to 2. That is not a regression in the sense that matters; the judge is less precise but safer. Conversely, exact match rising to 78% while severe misses rise from 5 to 14 means the prompt got better at the easy cases and worse at the ones that burn you. Decide the decision rule before you look, for example: ship if exact match does not drop by more than 5 points, within-one-level does not drop at all, and severe misses do not increase.

For per-item diagnosis, a small table helps. The Evidently regression-testing tutorial walks through exactly this pattern of before and after datasets and metrics that detect what moved (Evidently AI tutorial).

Two rows moving the same direction, toward leniency on vague copy, tells you more than an aggregate ever will.

Run-to-run variance matters too. If your judge is sampling at nonzero temperature, run the set three times and report the spread; a difference smaller than that spread is noise. The OpenAI forum thread on prompt regressions in production is a good reminder that "it worked for months and then moved" is a normal failure mode, not an exotic one (community discussion).

A regression set rots in predictable ways. Guard against each one.

Rubric drift. Someone edits the rubric in the prompt but not in the set, and now the labels and the judge are answering different questions. Fix: version the rubric and note the version in every run record.

Leakage. You write new headlines that resemble rows in the set because the team has memorized them. Fix: keep the set private to the evaluation workflow and refresh with new, unseen archive rows on a schedule.

Overfitting. You tune the prompt against the same 300 rows so many times that its score on them stops meaning anything. Fix: hold back a lockbox of roughly 20 to 30% of rows that you look at only when deciding to ship, and rotate a portion of the working set out over time.

Label decay. The metric that defines success changes (a channel shifts from clicks to reads) and the labels no longer reflect what you want. Fix: relabel affected rows and bump the set version rather than quietly overwriting them.

Selection bias in sampling. If you only pull your best-performing pieces, the set has no hard negatives and the judge looks better than it is. Sample deliberately across placements and performance bands.

Single-rater labels. One person's reading of a rubric becomes ground truth. Fix: require two raters on every new row and reconcile disagreements out loud.

None of this needs heavy tooling. A spreadsheet, a script that calls your judge over the JSON file, and a results table will do. What it does need is the discipline to run the set before every prompt or model change, not after an incident.

When the loop becomes routine and you want to run the same headline judge across several model families without rewriting client code for each one, Quanta Inference exposes text and embedding models behind one OpenAI-compatible API key, which keeps the regression harness simple. And when the harder question is not "did the score move" but "what should we change about the whole content operation," an expert engagement through https://quantaintelligence.ai/agents can take the archive, the rubric and the drift report and turn them into a revised workflow.