Recording Rater Scores Without Rater Names

Six fields carry almost all of the analytical weight. Everything else is convenience.

A seventh field is worth adding if you can: rubric_version. Rubric wording changes behavior. Guidance on evaluation rubrics is explicit that vague criteria are the main source of noisy evaluations, and that a leveled rubric with worked examples localizes disagreement instead of hiding it Arize's rubric glossary. If the rubric text itself changes between scoring rounds, the score table must say which text applied, or you will attribute a rubric rewrite to rater inconsistency.

What you must not store in this table: rater name, email, employee ID, IP address, device fingerprint, or free-text notes that name a colleague. Free-text rationales are valuable, but they leak identity through writing style and through references to prior conversations. Keep rationales in a separate table keyed to the score row if you need them, and apply the same retention rule as the map.

Here is a minimal definition, with a unique constraint that prevents the most common data bug in blind review, a rater silently overwriting their own earlier score:

CREATE TABLE rater_scores (
  score_id        BIGSERIAL PRIMARY KEY,
  study_id        TEXT NOT NULL,
  item_id         TEXT NOT NULL,
  rater_pseudonym TEXT NOT NULL,
  score           NUMERIC NOT NULL,
  condition       TEXT NOT NULL,
  rubric_version  TEXT NOT NULL,
  scored_at       TIMESTAMPTZ NOT NULL DEFAULT now(),
  UNIQUE (study_id, item_id, rater_pseudonym, condition, rubric_version)
);

The unique constraint enforces one score per pseudonym per item per condition per rubric version. If you want to allow revisions, add a superseded_at column and filter on it rather than deleting rows, because a deleted row is a lost data point and an audit gap.

A pseudonym is only useful if it is stable within a study and unlinkable across studies. Two designs work:

Per-study random pseudonym. The administrator generates a random string (a UUID or a salted hash) for each rater when the study opens and records the mapping in a separate table. The advantage is that the same person scoring in two studies gets two different pseudonyms, so nobody can stitch a rater's behavior together across studies from the score table alone. The cost is that you cannot aggregate a rater's calibration history across studies without the map.

Per-person random pseudonym. The administrator generates one random string per person, reused across all studies. This lets you ask "has this pseudonym improved since the last calibration round" without touching identity, which is genuinely useful for calibration policy. The cost is that the pseudonym becomes a persistent identifier, which is a privacy risk if it ever leaks, and a person's full scoring history becomes available to anyone with score table access.

The decision rule: use per-person pseudonyms when you have an explicit calibration program with a documented re-certification cycle, because the cross-study history is the whole point. Use per-study pseudonyms when the score table will be shared with people outside the core team, because cross-study linkage is the thing you least want them to have.

Either way, generate the pseudonym with a cryptographic random source or a keyed hash. Do not derive it from the name with an unsalted hash, since names are a small enough space that an unkeyed hash is reversible by dictionary attack.

import secrets

def new_pseudonym(prefix: str = "r") -> str:
    # 128 bits of randomness, hex encoded; no derivation from identity
    return f"{prefix}_{secrets.token_hex(16)}"

The map table holds the only link between a pseudonym and a person, and it should be structurally harder to reach than the score table. Concretely:

• Put the map in its own table with no foreign key from the score table pointing at it. A foreign key invites a join, and a join invites accidental exposure in an analyst's query results.
• Grant SELECT on the score table to the analysis role. Grant nothing on the map to that role.
• Grant the map to a single administrative role, and log every read of it.
• Consider a separate schema or a separate database. A separate database makes the boundary obvious in code review.
• Encrypt the name column at rest with a key held outside the database, so a database backup alone does not expose it.

A worked example shows why the split survives analysis. Suppose a study has 3 raters scoring 40 items each, all on the same condition, using the same rubric version. The score table has 120 rows and 3 distinct pseudonyms. An analyst can compute agreement without the map at all:

• Pairwise percentage agreement between each pair of pseudonyms: 3 pairs.
• Cohen's kappa for each pair, which corrects for chance agreement, as described in the standard kappa formula and its interpretation bands the kappasizeci documentation.
• An intraclass correlation across all 120 rows if scores are treated as continuous.
• Mean score per pseudonym, and the spread, which flags a systematically lenient rater.

None of those steps requires a name. The map is needed only when you want to route a coaching conversation to the right person, which is an administrative action with its own justification and log entry.

If you later delete the map entirely, every calculation above still runs. The pseudonym column is still there, still groups rows correctly, and still distinguishes scorer A from scorer B. You have lost the ability to say who A is, not the ability to say that A is an outlier.

The retention sequence should be: scores kept longest, map kept shortest, and the map deleted while the scores remain. This is the reverse of the intuitive order and it is the whole point of the design.

Retention practice in research settings already distinguishes between data that must be preserved for reproducibility and data that must be minimized for privacy. Cornell's research data retention policy frames preservation around the requirement that data be kept in sufficient detail to respond to questions about accuracy, authenticity, and compliance Cornell Policy 4.21. The same policy separately commits the institution to safeguarding confidential and identifiable data. Those two commitments pull in opposite directions, and the split-table design is how you satisfy both: keep the analyzable record, drop the identifier.

Survey platforms implement exactly this pattern. Qualtrics offers both a deletion policy and a separate anonymization policy, where you choose how long after submission specific identifying information is removed from the response Qualtrics retention policies. Note also the operational warning in that documentation: retention rules are retroactive and deleted responses cannot be retrieved. Build the rule with a dry-run mode first.

A concrete schedule for a typical blind review:

The trigger for map deletion should be an event, not only a clock: the moment the final report is accepted and no coaching action is pending. A calendar rule alone will either delete too early (you still need to tell someone their calibration slipped) or too late (the map sits around for years because nobody remembers it exists). Use both: an event trigger with a hard calendar cap as a backstop.

Implement it so the cap fires even if the event never does:

-- backstop: delete map rows for studies whose report was accepted
-- more than 180 days ago, or which are older than 3 years regardless
DELETE FROM rater_map
WHERE study_id IN (
  SELECT study_id FROM studies
  WHERE (report_accepted_at IS NOT NULL
         AND report_accepted_at < now() - INTERVAL '180 days')
     OR opened_at < now() - INTERVAL '3 years'
);

The reproducibility test for this schema is simple: hand the score table, and nothing else, to an analyst who has never seen the study, and ask for the agreement statistic. If they can produce it, the schema works. If they need the map, the design has failed.

Two practical checks make that test pass:

Check that pseudonyms are complete and stable. Count distinct pseudonyms per study per condition and compare to the number of raters you expected. A missing pseudonym usually means a rater submitted under two identifiers or a submission failed, both of which distort agreement.

Check that the item set is balanced. Agreement statistics assume raters scored overlapping items. If pseudonym A scored items 1 to 40 and pseudonym B scored items 30 to 70, the overlapping set is 11 items, and an agreement coefficient computed on the full set is meaningless. Store the item ID, not just a count, so this is checkable.

There is a further payoff: once scores and pseudonyms are in a stable table, you can add automated graders as additional pseudonyms rather than as a separate system. That is useful when you want to compare a model's scores against human scores on the same rubric. Microsoft's rubric reference guide formalizes this comparison with an alignment formula, alignment = 100% * (1 - |AI - Human| / 4) on a 1 to 5 scale, plus a directional bias check that counts how often the AI grades higher or lower than the human the Copilot Studio rubric guide. Both calculations need only paired scores. Log the model as a pseudonym with a condition of model, and the same table supports the comparison without a second pipeline.

The same logic applies to variance control. If graders themselves are stochastic, running multiple grading passes and aggregating reduces noise, and frequent disagreement on a single criterion is a signal that the criterion needs rewriting rather than more passes Hebbia's hybrid evaluation writeup. A schema with rater_pseudonym and condition already supports this: each pass is a row, and the pseudonym distinguishes passes by the same grader.

If your graders include a model, add one field and keep the rest identical: a judge_config column holding a hash of the judge instructions and settings. Judges return different scores across runs with consistent inputs, so the config hash is how you separate "the judge changed" from "the items changed."

Keep human and model scores in the same table, distinguished by condition or a rater_kind column, and never overwrite a human score with a model score. Paired analysis depends on both being present and separately identifiable. And resist the temptation to delete human rows once a judge is calibrated: the calibration only stays meaningful while the anchor scores remain.

Studies that generate a lot of scored items across many model backends tend to accumulate conditions quickly. Running the scoring calls through a single gateway keeps the score rows comparable, because one API key reaches text, image, and audio models without a per-provider integration, which is the setup Quanta Inference offers at https://quantaintelligence.ai/inference. Whatever backend you use, the schema rule holds: record which condition produced each score.