Prompting the Cheap Tier to Say When It Is Unsure

The prompt asks for three things in a fixed order, with the answer last. Reason first forces the model to commit to an interpretation. Band second forces a number before it has drafted text it might defend. Answer last gives you something to score.

You will answer a question about {task_domain}.

Reply with exactly three lines and nothing else.

Line 1, REASON: one sentence explaining the single most important
factor in your answer.

Line 2, CONFIDENCE: a band from this list only:
  0-50%, 50-60%, 60-70%, 70-80%, 80-90%, 90-100%
This means the chance your answer on line 3 is correct.

Line 3, ANSWER: your answer, as short as the task allows.

Four design choices carry most of the weight.

The band is coarse and the list is closed. Ten-point steps give six options. Finer steps invite meaningless precision, and a closed list makes parsing trivial and comparison across items fair. If the model can emit "73 percent," you have to decide what a 73 means, and so does every downstream consumer.

The meaning is defined in the prompt. The sentence "This means the chance your answer on line 3 is correct" is doing real work. Without it, models interpret confidence as fluency, as familiarity with the topic, or as certainty about the reasoning path. Define it once, in the system prompt, and keep the wording fixed across every call you ever score. Do not vary it between evaluation and production.

The order is reason, band, answer. Asking for the band before the answer keeps the model from anchoring on text it has already written. Asking for a short reason first gives the band something concrete to rest on without spending many tokens.

The rubric stays hidden. Never show the model your labeled examples, your target accuracy per band, or your acceptance thresholds. A model that can see the rubric will produce bands that match it, and the signal becomes a reflection of your instructions rather than the model's state.

Keep a fixed output budget. Use the output-token setting your endpoint supports, since Chat Completions uses max_completion_tokens and the Responses API uses max_output_tokens, per OpenAI's token documentation. The three-line format is short, so a modest cap protects you from a model that decides to write an essay instead of a band.

Fewer bands mean more items per bucket, which means tighter estimates. More bands give finer routing decisions but need more labeled data to validate. For a first pass, six bands over 300 items puts roughly 50 items per bucket if the bands are used evenly, and they will not be. Expect the middle and high bands to dominate.

A binary scheme is the most honest starting point if you only need an escalation trigger. It throws away information, but it is easy to hit 100 to 200 items per bucket, and the decision rule is trivial: escalate on low. The six-band scheme is the default recommendation because it lets you catch the specific failure that matters, a high band that is not accurate. Once you know your buckets, you can collapse them into a binary rule for production.

You need labeled items in the same distribution as production traffic, with a known correct answer and a scorer that decides correct or incorrect without consulting the model. 200 to 500 items is the working range. Below 200, a bucket with 20 items cannot distinguish 60 percent accuracy from 80 percent.

The procedure:

1. Freeze the prompt, the band list, and the model configuration. Record a prompt version identifier on every call.
2. Run every labeled item through the exact production path and store the parsed reason, band, answer, correctness, token counts, and prompt version.
3. Bucket by band. For each bucket, compute accuracy as correct divided by total, and the midpoint of the claimed band.
4. Compare. A well-behaved signal has accuracy inside or close to the claimed range in every bucket, and it rises monotonically as the band rises.

Track token usage at the same time, because the confidence check has a cost. Token counts differ from word counts and can differ across models for identical text, and reasoning tokens are billed as output even though you never see them, per OpenAI's token documentation. A 200-item validation run is cheap, but the per-request overhead of reason and band lines is permanent, so measure it. Guidance on attributing spend per request, feature, and prompt version is in this cost-tracking playbook and this guide to tracking token cost across workflows, both of which treat a prompt version tag as the minimum for comparing one prompt against another.

A small script does the arithmetic:

from collections import defaultdict

# rows: {"band": "70-80", "correct": True, "prompt_version": "v3"}
def reliability(rows):
    buckets = defaultdict(lambda: {"n": 0, "correct": 0})
    for r in rows:
        b = buckets[r["band"]]
        b["n"] += 1
        b["correct"] += int(r["correct"])
    out = []
    for band, b in sorted(buckets.items(),
                          key=lambda kv: float(kv[0].split("-")[0])):
        out.append({
            "band": band,
            "n": b["n"],
            "accuracy": b["correct"] / b["n"],
            "claimed_mid": (float(band.split("-")[0]) +
                            float(band.split("-")[1].rstrip("%"))) / 200,
        })
    return out

Run it per prompt version. If you change the wording of the confidence instruction, you have a new instrument and you need to validate it again.

Here is a worked example with 400 items, six bands, and a hypothetical outcome.

The headline is the 70-80 percent bucket: it claims at least 70 and delivers 55. That bucket alone holds 118 items, nearly 30 percent of traffic, and it is the bucket most production rules would trust. A rule that accepts anything at 70 percent or above would take 277 items, 69 percent of traffic, at an overall accuracy of about 0.72, well below the 0.75 the model's high confidence implies.

Two decisions show up in this table. The first is whether the signal is usable at all. Monotonicity holds here: each higher band is more accurate than the one below except for the 70-80 dip, which is exactly the kind of anomaly that kills a naive threshold. The second is how to fix it. Either widen the bands, merging 70-80 into a 70-100 bucket that reports 0.73 and is honestly labeled as such, or recalibrate by applying an offset learned from this table, or stop trusting the band and gate on something else. Recalibration is the standard move for overconfident intervals: a conformal approach restored accurate 99 percent coverage and cut the interval score by 54 percent in the FermiEval study.

The most common failure is not the dip. It is the flat line. If the 60-70, 70-80, and 80-90 buckets all land between 0.68 and 0.72, the model has one confidence level dressed in three costumes, and no threshold on that number will separate the easy from the hard. In that case, stop scoring the band and change the input to the decision: escalate based on the reason line's content, on retrieval quality if the task is grounded, or on a second cheap check.

The band is useless until it gates something. The simplest rule uses two thresholds and the calibrated accuracy table, not the stated band alone.

• Accept when the calibrated accuracy for the item's band meets your quality bar. If your bar is 0.85 and the 80-90 bucket is calibrated to 0.78, that bucket does not qualify even though the model claimed 85.
• Escalate to a stronger model when the calibrated accuracy falls below the bar but above a floor, so you spend the extra tokens only where they might buy a correct answer.
• Abstain or hand off to a human when the calibrated accuracy is below the floor. An explicit "I don't know" that triggers escalation costs less than a confident wrong answer; early abstention research achieved a 13 percent cost reduction and a 5 percent error reduction while accepting a 4.1 point increase in abstention rate, as reported in the routing and cascades writeup.

Measure in cost per successful outcome, not cost per token, because two models with the same accuracy can differ enormously in tokens spent to get there. Cost per outcome is the metric that ties spend to whether the work was good, per Splunk's writeup on measuring AI. Worked example: a cheap call costs a fraction of a cent, an escalation costs several times more, and a human review costs dollars. If the confidence band lets you escalate 20 percent of a 10,000-item batch instead of sending all of it to the strong model, the saving is roughly 80 percent of the difference in per-item price, before counting the reason and band lines you added to every cheap call.

The design also interacts with how you pick the model. If the confidence check shows the cheap tier is trustworthy on 60 percent of your traffic, routing is now a data-backed decision. Unified access across providers through one API key, priced in dollars, removes the plumbing so you can swap tiers and rerun the same validation, which is the point of Quanta Inference.

Self-reported confidence is one signal among several. Pick based on what you can measure and what your API exposes.

The practical order is: start with the stated band because it costs almost nothing to add and is easy to throw away. If it fails validation, add answer consistency for the ambiguous middle band only, and if that is still weak, train a small classifier on the reason text using your labeled set. Failure detection research found that aggregating multiple steered confidence estimates outperformed single confidence signals on calibration and failure detection, per the SteeringConf paper, and an overview of the method is in this alphaXiv summary. If you want someone to run this loop with you, a consulting call is the fastest path, and if you are building the validation harness inside a coding agent, the relevant Quanta skills can be installed into the agent directly.