Development

Automated judges and calibration

• Evaluating Prompt Candidates Without a Human Judge
• Scoring an Automated Judge on the Golden Set First
• A Judge's Agreement Score: When to Trust It and When to Retrain
• A Relevance Rubric an Automated Judge Can Apply
• Calibrating a Judge Against a Cheap Model: Agreement and Bias
• Calibrating a Judge Against Human Labels: Size and Agreement
• Calibrating an Automated Judge Against Human Labels
• Calibration Before Thresholds: Making a Score Mean What It Says
• Confidence Scores an Agent Can Trust: Calibration
• Judging Model Output Without a Human: Rubrics and Calibration
• Recording a Rubric Exception When a Reviewer Overrides the Judge
• Telling a Real Quality Drop from Judge Noise on a Small Sample
• When a Human Judge and a Model Judge Disagree: A Tie-Break Rule
• When the Judge Disagrees With the Assertion