Anonymized Data for Model Evaluation: Building a Held-Out Set

Anonymization for eval has a different bar than anonymization for developer sandboxes. A developer sandbox needs working joins. An eval corpus needs the distribution preserved, because a model's accuracy depends on input shape, length, language mix and formatting.

Deterministic transformation is the core technique. Hash each identifier with a secret salt (SHA-256 or SHA-3) so the same input always produces the same output, preserving foreign keys across tables while staying irreversible Xata on anonymization. For free text, keep the structure and replace only the entities: names, emails, phone numbers, account identifiers. Keep email domains if your prompt or model branches on them, and mask only the local part.

Where formats are validated downstream, format-preserving encryption (FPE) keeps the value in shape. 123-45-6789 can be encrypted to a valid-looking 529-38-1746, which preserves length and checksum-adjacent rules without exposing the original Tonic on format-preserving encryption. FPE also supports determinism, so the same input yields the same ciphertext, which matters when your join key is anonymous but still referenced.

A checklist before you call a snapshot clean:

• Every free-text field that can hold PII passed a regex and an entity-recognition pass.
• Identifiers are hashed with a salt stored outside the repository.
• Row counts match the source subset after filtering.
• A human spot-checked 50 random rows against the source.
• The anonymization script is in version control, keyed to the snapshot id.

One warning that practitioners learn the hard way: do not null out everything. If email masking turns every value into an empty string, the model sees a distribution it will never see in production and your metrics become meaningless. Preserve the shape, replace the content.

Before you label anything, write down what a single eval record contains. Most teams converge on a JSONL format where each line has an input, an optional context, an expected output or an expected behavior, tags and a note. The qaskills guide to golden datasets shows this structure and stresses that many good cases specify expected behavior instead of one canonical string golden dataset guide.

{"id": "e-0001",
 "input": "...",
 "context": "...",
 "expected_behavior": "Must refuse and cite the refund window",
 "tags": ["policy", "edge", "refusal"],
 "note": "Seen in staging on 14 requests; model previously invented 60 days."}

The label schema itself is the part teams rush. Write a rubric with three things: the label set, one example per label, and a decision rule for the ambiguous middle. If your task is classification, decide whether "partially correct" is its own label or a failure. If your task is generation, decide whether a judge scores with a binary pass/fail or a 1 to 5 scale and what each number means in one sentence.

Then appoint one gold owner. Musubi's guide to content moderation datasets calls for a single gold owner with final authority on all labels and documentation of the reasoning, not just the label Musubi on golden datasets. That is the decision rule that ends the standoff when two labelers disagree on a borderline case: the owner decides, the rubric gets amended, and the change is versioned.

A random sample of your staging snapshot is a fine starting point and a bad eval set. It is dominated by the common case, so a model can score 90 percent while failing on every input that a human reviewer would call difficult. Build stratified coverage instead.

Stratify by the dimensions that actually change model behavior: input length buckets, language, presence of numbers or dates, presence of out-of-scope requests, and known high-cost categories. Then add adversarial slices that a random sample rarely contains:

• False-positive traps. Inputs that look like a category but are not (a complaint that reads like a refund request).
• Near-duplicates with one changed token. Names, dates or negations flipped, which exposes brittle prompts.
• Truncation and encoding edges. Emoji, right-to-left text, pasted logs.
• Nulls and junk from real migrations, which synthetic data misses Xata.

Sizing. The moderation guide gives concrete starting points: 30 to 50 examples for a single policy area, at least 100 for high-stakes policies, and 100 to 300 for a comprehensive multi-policy set, with a composition roughly of 30 to 40 percent clear violations, 40 to 50 percent clearly safe, 10 to 15 percent false-positive traps and 10 to 15 percent hard edges Musubi. Those ratios shift if you are testing recall rather than precision: a recall-focused set runs 60 to 70 percent violations.

Do the arithmetic before you commit. With 100 cases, one flipped label moves accuracy by a full percentage point. With 40 cases, two and a half points. If your model and prompt variants differ by 2 points, a 40-case set cannot tell them apart, so either size up or report the confidence interval alongside the score.

Keep the eval set out of training and few-shot examples entirely. Musubi puts it plainly: using the golden set for fine-tuning defeats its purpose as an independent test, so keep training and evaluation data separate Musubi. Audit your prompt for leaked eval strings before every run.

Treat the corpus like code. It lives in a repository, changes through pull requests, and every release has a version that results reference. A flat folder works:

eval-corpus/
  v1.0/
    manifest.json      # snapshot_id, row count, checksum, schema version
    labels.csv         # id, label, rubric_version, labeler, note
    cases.jsonl        # input, context, expected_behavior, tags
    rubric.md          # the written decision rules
  CHANGELOG.md

Three rules make versioning useful:

• Never edit a released version. Fix a label by creating v1.1, and keep v1.0 for trend lines.
• Pin the run. Every eval result records the corpus version, the prompt hash and the model identifier, so a score is always reproducible.
• Lock the schema. If the label schema changes from binary to graded, that is a major version bump, and old scores are not comparable.

Langfuse's guidance on golden datasets matches this: freeze version 1.0 with complete metadata and never change it, and create new versions when policies change so you can still track trends Langfuse on golden datasets. The manifest checksum is the cheap insurance: if someone edits a file in place, the checksum fails and the run refuses to start.

For storage, keep the corpus out of the same bucket as raw staging exports. Different access rules, different retention, and ideally different credentials. The eval set is anonymized, but it is still a copy of user-adjacent data.

The hardest question is when to rebuild the corpus. Rebuild too often and last week's score is not comparable to this week's. Never rebuild and the corpus slowly diverges from live traffic until the model is excellent at examples users stopped sending.

The answer is a written refresh rule with named triggers and a named owner. Only a trigger, not a calendar, opens a new version.

The drift row is where most teams need help. You cannot pick a threshold without knowing how much your data normally moves, so run drift detection on historical data first to learn the dynamics and choose thresholds that avoid both false alarms and silent decay Evidently on historical data drift. Research on dataset stability benchmarking makes the same point from a different angle: datasets themselves drift, and monitoring a dataset against concept drift can tell you which parts are weak before you retrain arXiv dataset stability benchmark.

A worked example of the decision rule. Say your corpus is 120 cases and your current model scores 0.87 pass rate. A new prompt scores 0.89. With 120 cases, that gap is about two and a half cases, which is well inside normal labeler noise. The rule should say: do not ship on a two-point gain unless the confidence intervals do not overlap, or unless the gain is concentrated in a slice you specifically set out to fix (for example, adversarial refusals pass 6 of 10 instead of 3 of 10). That converts a coin-flip decision into a stated policy.

Two more rules that prevent trouble:

• Freeze during evaluation. No one edits the corpus while a comparison is running. Edits land in the next version.
• Budget the labeling. Refresh means human labeling hours. Estimate them up front; a 120-case refresh at a few minutes per case is a real cost, so trigger it deliberately.

An eval corpus that lives in a spreadsheet changes nothing. To change decisions, it has to run automatically and its result has to gate the release.

The pattern is a regression gate in CI. Every prompt change and every model swap runs the corpus, produces a pass rate per slice and a delta against the last released version, and blocks if a named slice regresses beyond tolerance. qaskills describes wiring the golden set up as a CI regression gate golden dataset guide, which is the point at which the corpus starts paying for itself.

Pair the gate with a staged rollout. Feature flags let teams decouple deployment from release and expose a change to a small slice of traffic first, as little as 1 percent, instead of everyone at once LaunchDarkly on feature flags. Datadog's overview makes the same case: flags let you release incrementally and disable a feature instantly for affected cohorts Datadog feature flags guide. The corpus tells you whether the change is worth shipping; the flag decides how much traffic sees it.

If you are routing several models behind one application, keeping an OpenAI-compatible interface across text, image and embedding models removes the per-provider plumbing from the eval loop. Quanta Inference (https://quantaintelligence.ai/inference) exposes every model kind through one API key, which means one eval harness and one set of corpus runs regardless of which backend answers.

Finally, feed the corpus back into the product. The adversarial cases and incidents you capture are the same examples that teach teammates what "good" looks like. If your team builds internal tooling or sites alongside the model work, a specialist at Quanta (https://quantaintelligence.ai/) can take a described app or plan and build it with you, eval dashboard included.