How to Blind a Scoring Sheet So Labels Do Not Leak In

The most common error is blinding the input but leaving the field in the rubric. A rubric that mentions the field, even as a scoring criterion, tells the rater to attend to it. Rubrics are explicit sets of criteria with defined performance levels, scores, and descriptors, and their whole purpose is to make scoring objective and consistent (NIU Center for Innovative Teaching and Learning). That same structure makes a contaminated rubric a reliable channel for leakage.

Treat three artifacts as separate outputs of one process:

The NC State teaching resource lays out the rubric types (holistic, analytic, single-point) and their tradeoffs, which matters here: an analytic rubric scores criteria separately, so you can inspect each criterion line for a leaked field (NC State). A single-point rubric concentrates on the proficient level, which is easier to harden, but it also gives the rater more room to interpret, and that freedom can reintroduce the field you removed from the words. Pick the type that matches how much inspection you are prepared to do per criterion.

Keep the protected field in a locked side table keyed to an internal ID. You need it for the adverse impact analysis. You just do not need it in anything a human or model rater touches.

Removing the demographics column is the easy part. Proxies are the hard part, and they sit inside free text. A resume or profile carries name, school, graduation year, address, club memberships, language choices, and career-gap phrasing, each of which can correlate with a protected characteristic. The four-fifths methodology warns that facially neutral criteria can still produce disparity, so stripping the explicit column is necessary but not sufficient (Pin).

A stripping pass that works in practice:

1. Replace the name and any personal identifier with a stable token, and store the mapping separately.
2. Remove dates of birth and graduation years, keeping only years of experience where job-relevant.
3. Generalize institution and employer names to role type, or remove them where they are not job-relevant.
4. Scrub addresses, languages, memberships, and any free-text line that names a group.
5. Re-run the probe on the stripped record and confirm it no longer moves the score.

Steps 1 through 4 are mechanical. Step 5 is the one people skip, and it is the only step that proves the stripping worked. One practical shortcut: require every kept field to state its job-relatedness in the rubric. If a field cannot be tied to a criterion, it does not go into the rater input. That single rule catches most accidental proxies before they reach a rater.

The Medium account of resume screening points at the failure mode when the field survives: an internal tool that trained on ten years of male-dominated hiring data learned to penalize resumes mentioning women's activities (Medium). The fix is not a better model. It is to stop the signal from reaching the scorer in the first place.

A blind rubric says what good performance looks like without saying who tends to produce it. The NIU guide's rubric elements (criteria, levels, scores, descriptors) give you a checklist to iterate line by line: while reviewing each descriptor, ask whether it implies the protected field. The Lincoln University rubric quality guide draws the same line, requiring that criteria cover essential features and leave out trivial or unrelated features (Lincoln University). A leaked field is an unrelated feature by definition.

Compare a leaked criterion with a blind one:

The replacement on each row is measurable, so a human and a model rater can agree on it. That is the point of a rubric: reducing subjectivity through explicit descriptors (NIU). A criterion no two raters can score identically is not measuring the job, and it is usually where the field hides.

Run a plain-language check on the final rubric: read every descriptor out loud and ask whether a reader unfamiliar with your organization would guess the protected field from it. If yes, rewrite it. Brooks and Caliskan note that many debiasing approaches in hiring do not address the retrieval and ranking step where the field enters, which is why the rubric must be tested, not just declared clean (Brookings).

Blinding is a claim until you test it. The null-field control does that in one step: take the protected field, replace every value with a constant or shuffle it across records at random, and re-run the exact same scoring round. If the score distribution moves, something in the pipeline is still reading the field or a proxy of it.

A worked example with numbers:

• In round 1, the probe shows the field moves the mean score by 0.8 points on a 5-point rubric.
• In round 2, you strip the field and its proxies, blind the rubric, and re-score.
• The null-field control in round 2 changes the mean by 0.05 points, within scoring noise. The blinding held.
• If instead the control still shifts the mean by 0.6 points, the leak is in the rubric wording or a proxy you missed, and you iterate.

The two-sided test matters. Turnitin's own guidance says a per-document AI percentage above 0 and below 20 percent is suppressed because false positives cluster below 20 percent (Turnitin). The lesson generalizes: define your noise band before you run the control, and treat movement inside the band as no movement. Run the control on a fixed sample, not the full corpus, so you can iterate quickly.

Document the control result next to the blinding steps. "Field removed from input, labels, and rubric; null-field control moved mean by X; noise band defined as Y" is a sentence an auditor can check, and it is the difference between a claim and evidence.

The audit trail is not paperwork. It is what lets a second reviewer reproduce the round without re-deriving your choices. The four-fifths methodology and the adverse impact literature both depend on a documented record of the pool, the selection logic, and the rate calculation across groups (BambooHR). Blinding is the input to that record.

A blinding record should capture:

• The field or fields blinded, and the probe result that triggered the blinding.
• The exact stripping rules applied to the rater input, with a before-and-after example.
• The rubric version with a note on which descriptors were rewritten and why.
• The null-field control method, sample, movement observed, and pre-registered noise band.
• Who ran the round, when, and who reviewed it.

That last line is the one most teams skip and the one a human reviewer will ask for first. Two raters who apply the same blinded record to the same inputs should get comparable scores; if they do not, the rubric is not clear enough, and the Lincoln guidance on clearly defined performance levels is the point of reference (Lincoln University).

Documenting also protects the model-choice decision. Building a scoring pipeline in-house looks cheap on the invoice, but the Metaview breakdown of build-vs-buy shows what not-on-the-invoice maintenance, integration, and drift handling actually cost over a lifecycle (Metaview). A blind, documented round is something you can hand to a vendor or an internal team as a spec, so whichever way you build, you are measuring the same thing. Quanta's build service is one route to a documented, blinded pipeline if you would rather hand the build to a specialist than staff it internally.

Expect score levels to fall in absolute terms. Removing the field removes whatever advantage it conferred, and the mean will shift. That is the blinding working, not a regression. What you should watch is the spread across groups: if the selection rate ratio moves toward 1.0, the field was doing the work; if it stays flat, the field was not the main driver and you should look at the next-most influential variable in the probe.

Keep the protected field in the side table and re-run the four-fifths calculation on the blinded round. The rule flags a selection rate below 80 percent of the highest group's rate, and the EEOC frames it as the first stage of investigation, not proof of discrimination (Berkshire Associates). A blinded round that still audits out as biased gives you a much stronger basis for changing the criteria than an unblinded one, because you have removed the obvious explanation.

The same logic applies whether the rater is a human panel or a model. The Fisher Phillips blueprint on AI resume screening recommends structured, multi-measure review and ongoing monitoring, which assumes a clean input and a clean rubric (Fisher Phillips). Blinding is the precondition for that monitoring to mean anything. For teams running several model types against the same blinded sheet, a single key for text, image, and embedding models removes a moving part: Quanta Inference gives one OpenAI-compatible API for multiple model families (Quanta Inference), so a blinded round can be re-run against a different model without rewriting the harness.