From Shadow Score to Rollout Decision: Rubric to Go/No-Go

The most common failure in converting rubric scores is mixing scales. One dimension is scored 1 to 5, another is a pass or fail, a third is a percentage. Normalize everything to 0 to 100 before you weight anything.

Microsoft's rubric reference uses a standard 5-point scale from "exemplary" to "needs improvement" and defines each level with observable conditions the Copilot Studio rubric guide. Map that scale linearly:

def to_hundred(grade, low=1, high=5):
    """Convert a 1-5 rubric grade to a 0-100 dimension score."""
    return (grade - low) / (high - low) * 100

for g in [1, 2, 3, 4, 5]:
    print(g, round(to_hundred(g), 1))

That prints 0, 25, 50, 75, 100. A pass/fail dimension becomes 0 or 100. A share of cases passing a deterministic check is already 0 to 100.

Two cautions. First, prefer few levels. Rubric guidance warns that ten-point scales are almost always a mistake because nobody can articulate the difference between a 6 and a 7, and both humans and judges compress scores toward the middle the Arize glossary. Second, keep one criterion per failure mode. Hebbia's team reports that atomic criteria make graders more consistent and failures easier to debug, and that binary criteria that pass or fail together are probably measuring the same capability and should be merged the Hebbia framework. Atomic criteria give you clean dimensions to weight.

Keep total dimensions small. A short set of clear dimensions applied consistently tends to produce cleaner signal than a long checklist that leaves room for interpretation the Label Studio guide. Six to eight weighted dimensions is enough for most pilots.

Weights encode risk. In a low-risk internal tool, tone and completeness might carry real weight. In a high-risk workflow, safety and grounding dominate and tone barely moves the number. Set weights per risk tier once and freeze them before scoring, so nobody can reweight after seeing results.

A workable default set of dimensions, drawn from the three evaluation pillars of agent success, process and trajectory, and trust and safety the Google Cloud guide:

Compute the rollout score as a plain weighted sum:

weights = {"success": 0.25, "trajectory": 0.20, "grounding": 0.20,
           "safety": 0.20, "tone": 0.05, "cost": 0.10}
scores = {"success": 85, "trajectory": 72, "grounding": 90,
          "safety": 88, "tone": 60, "cost": 80}

rollout_score = sum(weights[k] * scores[k] for k in weights)
print(round(rollout_score, 1))

That agent scores 82.3, which passes an 80 bar. Note what the arithmetic hides: tone sits at 60. This is why the pass bar needs floor rules, covered next.

A weighted average lets a strong dimension rescue a weak one. That is fine for an internal tool and unacceptable for anything that touches money, health, or legal status. Every tier therefore needs two conditions: a composite bar and a per-dimension floor. The short answer above gives baselines, and here they are as a checklist you can copy into your decision doc.

The reason for the floor is visible in the worked example. A "helpfulness" bar of 80 hides a tone score of 60 entirely, and if the use case is a customer-facing agent that is exactly where complaints come from. The same asymmetry shows up in trajectory scoring. Splunk's framework notes that Agent S3 with GPT-5 reached about 78% pass@10 but only about 36% pass^10, meaning a system can look capable while being too inconsistent to ship the Splunk framework. Your composite score should be computed on a consistency basis, not a best-of run. Score every case with all outputs, and treat any critical failure as an automatic no-go regardless of average.

On the sample size: A/B guidance for agents argues that quality metrics can need 10,000+ trajectories per arm to separate real lift from stochasticity, especially for subjective properties the Maxim AI guide. Most pilots will not pay for that before launch, which is precisely why the offline test sizes above are modest and the large samples belong to the expansion stage, not the launch gate.

Two outcomes are not enough. A pilot that clears its bar and fails at scale wastes a quarter. A pilot that misses its bar narrowly and gets killed loses information. Use three verdicts with explicit triggers.

• Launch: composite at or above the tier bar, all floors met, and all evidence gates satisfied. Roll out to the defined initial scope only.
• Expand: the pilot has been live at the previous stage for the agreed observation window, the composite has held within agreed tolerance of the launch score, and no critical incident has occurred. This is where you grow traffic.
• Stop: composite below the tier bar, any floor breached, any critical failure, or an unresolved incident during the observation window.

The expand stage has a well-understood structure. Shadow deployment runs the new model alongside production on real traffic without serving its outputs to users, then compares shadow and champion outputs for accuracy, latency and distribution drift before promotion the Atlan guide. A common sequence is shadow, then canary at 1 percent, then 20, 50, and finally 100 percent, keeping the last version available for rollback the Qwak comparison. That is the motion behind an "expand" verdict: each stage has its own threshold, and a regression sends you back to the launch gate rather than to zero. The original shadow mode write-up makes the same point from the operations side: shadow mode tests both whether the system handles inputs as intended and whether it can handle the load the shadow mode guide.

A decision rule is only as good as the scores feeding it. Two failure modes will quietly break your threshold.

The first is judge drift. LLM judges agree with human reviewers a lot of the time, roughly 85% agreement in the widely cited study compared against about the same between two humans, but the wrong prompt or rubric makes them as flaky as the model under test the Confident AI guide. Google's guidance is explicit: before deployment, align the LLM judge to human evaluation by comparing judge output against the original manual human output the Google Cloud guide. Microsoft's formula makes the check arithmetic: alignment equals one minus the absolute gap between AI and human grade divided by the scale span, so 24 exact matches out of 30 cases is 80% alignment the Copilot Studio rubric guide. Report that number in the decision meeting. Dalow alignment means the composite score is not yet a decision input.

The second is judge nondeterminism. Running multiple grading passes and aggregating reduces noise, and if graders frequently disagree on a criterion, that criterion needs refinement rather than more passes the Hebbia framework. Splunk recommends running the same representative sample several times and reviewing borderline items that flip between runs the Splunk framework.

Finally, store scores with the case IDs and rationale. Disagreements are valuable because they show where the evaluation needs attention, and a recorded rationale plus adjudication keeps standards consistent over time the Label Studio guide. When you pick a model to run the judge or the agent under test, a single compatible endpoint for every model type saves you from re-plumbing the harness each time; Quanta Inference (https://quantaintelligence.ai/inference) exposes text, image, and embedding models through one OpenAI-compatible key.

The meeting is short if the inputs are ready. Run it as a decision, not a status update.

Roles. Four named seats: an evaluation owner who presents the scores and owns the harness, a product owner who defines the success statement and scope, a risk owner who can veto on their tier, and one independent reviewer who did not build the agent.

Inputs, circulated 24 hours ahead. The composite score with per-dimension breakdown, the pass bar and floors for the declared tier, judge-to-human alignment, the list of critical failures with case IDs, the human review notes from the live cases, and the proposed next-stage scope if the verdict is launch or expand.

Voting rule. The risk owner holds a veto; it applies to high and medium tiers and to any critical failure. Otherwise the verdict passes if at least three of the four seats agree with the score-based recommendation. If the composite and the recommendation disagree, the score wins and the meeting records why the recommendation was wrong. Every decision gets written down with the score snapshot, the tier, and the date; a later change in prompt, retrieval index, or model version voids the snapshot and requires a new measurement.

That last clause matters more than it looks. Standards should stay consistent while benchmarks stay relevant, and the mechanism is a versioned record of what was measured and when the Label Studio guide. Without it, teams inherit a "passed" label nobody can defend six months later.

If your team lacks the domain reviewers to sit in that meeting, a domain expert engagement can supply the judgment: Quanta Experts (https://quantaintelligence.ai/agents) has each expert read your context and hand back finished deliverables, which fits the ground-truth and calibration work better than a generic reviewer pool.