Using a Model to Score Headline Clarity Against a Rubric

Each check has to be answerable with yes or no from the headline plus the page context you supply. If a check needs judgment calls the model cannot resolve, tighten the definition until it can.

Two details matter. First, the number check is not only about digits. "Two days", "weekly" and "flat fee" all pass, because each gives the reader something concrete to picture. Second, the source check is often the weakest link on landing pages, and it connects directly to how much a page can be believed.

Attribution matters because credibility is a known usability factor on the web. Jakob Nielsen's original research found that users detested promotional writing style, calling it marketese, and that credibility improved with good writing and outbound links (NN/G on how users read on the web). Later eyetracking summed across more than 500 participants found scanning behavior unchanged across two decades (NN/G on how people read online), which is the reason a headline has to deliver its payload in the first few words a gaze lands on.

The cleverness flag deserves its own paragraph because it is the check people resist. Wordplay is not automatically bad. The flag should fire only when a reader could not restate the offer after reading the headline alone. A rubric item phrased that way gives the model a testable rule rather than a taste judgment. Copywriting guidance reaches the same conclusion from the practitioner side: never sacrifice clarity for cleverness, because a confused reader does not click (headline writing tips). Directness is also what ad platforms ask for: Google's guidance for Responsive Search Ads is to state the product or service in plain language, and "Buy Running Shoes Online" beats "Step Into Something New" (Coinis on Google ad headlines).

Feed the model the headline, the page's first paragraph (so it can tell what the offer actually is), and the audience. Then require a single JSON object. Constrain decoding to that schema if your API supports it, and set temperature low so repeat runs agree.

{
  "headline": "string",
  "entity":        {"pass": true,  "quote": "string", "reason": "string"},
  "verb":          {"pass": true,  "quote": "string", "reason": "string"},
  "number":        {"pass": true,  "quote": "string", "reason": "string"},
  "source":        {"pass": false, "quote": "",       "reason": "string"},
  "cleverness":    {"flagged": false, "reason": "string"},
  "clarity_score": 0,
  "rewrite_hint": "string"
}

The prompt that produces it:

You are scoring one marketing headline against a fixed rubric.

CONTEXT
Page first paragraph: {page_intro}
Audience: {audience}
Headline: {headline}

RULES
For each check, answer only from the headline text.
Give the exact substring that satisfies the check in "quote".
If no substring satisfies it, leave "quote" empty and say why in "reason".
Keep each "reason" to one sentence and do not restate the rule.

CHECKS
entity     : the headline names the product, company, category or person.
verb       : a verb states what the reader does or what the product does.
number     : a quantity, timeframe, price boundary or checkable claim appears.
source     : a claim is tied to a named source, study, customer, reviewer or standard.
cleverness : flagged true only if a reader could not restate the offer from the headline alone.

SCORING
Start at 10.
Subtract 2 for each failed check among entity, verb, number, source.
Subtract 1 if cleverness is flagged.
Never return a score above 6 when two or more checks fail.

OUTPUT
Return only the JSON object. No prose before or after.

The scoring weights encode a judgment call you should make deliberately: entity and verb failures cost more than a missing attribution, because a headline without a subject or an action is not a headline, it is a slogan. Tune the weights once, write them into the prompt, and leave them alone between runs so batch scores stay comparable.

Two ways to make the judge more reliable, both documented in evaluation practice, are worth adopting. Chain-of-thought prompting, where the judge writes a rationale before the verdict, reduces the tendency to hand out generous scores, and few-shot examples of headlines you have already scored anchor the scale (Confident AI's guide to model judges). The quote field in the schema does the same work more cheaply: a model that must copy the exact words cannot quietly credit an entity that is not there.

Take a small batch and run it by hand first. Here is a five-headline batch scored against the rubric above. The scores are this author's, produced by applying the arithmetic in the prompt.

Only one headline clears the bar. The interesting cases are rows three and five. Row three is perfectly clear and still scores 6, because it makes no specific claim and cites nobody; that is the headline a team ships when it is afraid of overpromising, and the rubric says it will not beat a page's alternative that does make a claim. Row five is the cleverness flag doing its job: the reader learns the metaphor and not the offer, and the score reflects that even though the verb check passes.

A worked batch like this gives you three things: an anchor set for few-shot examples, a sanity check on the arithmetic, and a quick read on which failures cluster. In the table, the number check fails in four of five headlines. That is a content problem, not a model problem, and the batch report should say so in one line at the top.

For larger sets, the calibration work is the same but the arithmetic shifts to agreement. Score twenty headlines by hand, run them through the model, and count how often the model's pass/fail matches yours per check. Research on a calibrated rubric framework reported RMS error under 0.5 on a 1 to 4 overall satisfaction scale, roughly a 2x improvement over an uncalibrated baseline, when many rubric answers were combined (LLM-Rubric paper). You do not need that machinery for headlines, but the lesson holds: the raw score improves when you combine several structured answers instead of trusting one.

Run these steps once per rubric version, then again whenever you change a check definition.

1. Hand-label twenty headlines you have already shipped or rejected, with a known reason for each decision. Include at least five you consider bad, so the judge sees the bottom of the scale.
2. Run the prompt on all twenty at low temperature.
3. Compare per check, not on total score. A judge that gets the total right for the wrong reasons will drift.
4. For every disagreement, read the model's reason. If the reason is defensible and your label was sloppy, fix the label. If the reason shows a misread of the rule, tighten the rule.
5. Add two or three of the hardest disagreements as few-shot examples in the prompt.
6. Re-run and record the agreement rate. Treat anything below about eight in ten on the number and source checks as a signal to tighten definitions.

LangChain's guidance on judge calibration makes the same point about ordering: collect human corrections, turn them into examples, and track agreement over time, because prompt iteration alone does not close the gap between a technically correct judge and a reliable one (LangChain on calibrating model judges).

Two practical protections belong in the batch job. Randomize or average out position when you compare two headline variants side by side, since judges favor the first or last option presented. And keep a small held-out set you never tune against, so you can tell whether the latest rubric edit generalized or just memorized the examples.

The score is a filter, not a verdict. A useful pattern for a content team is to route every headline through the judge before the page is built, and to publish only headlines that clear both the total threshold and the number check, because the number check is the one that most often exposes a page with nothing specific to say.

The score should also feed the page, not just the headline. Messaging match is the first landing page practice on Unbounce's list: the page a visitor lands on has to match the ad or promise that sent them there, and mismatches drive people away (Unbounce landing page best practices). A headline that fails the source check usually means the page also states claims without support, and the fix belongs in both places at once. Social proof is the cheapest source to add: connecting the claim to a named customer or a number from a study turns a fail into a pass, and testimonials outperform bare logos in at least one split test, with a 35% lift for the customer quote version (UserEvidence on social proof and landing page conversions).

If you want the batch to run across models rather than being pinned to one vendor, one API key that reaches text models from several providers removes the plumbing, and Quanta Inference is built for exactly that shape of job (https://quantaintelligence.ai/inference).