How to Test Whether a Resume Screener Ignores the School Name

Start with the scorecard, not the resumes. A role scorecard states the purpose of the role, the results the person is accountable for, the responsibilities, and the skills required, and it should exist before the role is posted (Stephen Lynch; Rhythm Systems). Every synthetic profile you write should be defensibly strong or defensibly weak against that scorecard, so that a school name is never the only thing carrying the score.

Practical construction rules:

• Pick a band of qualification, not a spectrum. Write profiles that should score in the middle-to-strong range if the screener works. Weak profiles tell you little about school sensitivity because the screener may correctly reject them for reasons unrelated to school.
• Vary the substantive content across profiles. Different job titles, industries, years, project types, and tool stacks. You want 20 to 30 profiles so that a school effect showing up on one profile is not a fluke of that profile's phrasing.
• Never include a real person's data. Fictional names, fictional employers, fictional achievements. This keeps the test outside the privacy perimeter of live candidates and lets you write freely.
• Write the school variants as four matched conditions. Elite (a well-known selective institution), mid-tier (a recognizable regional state school), unknown (a plausible but obscure institution), and blank (no education field at all, or a degree with institution omitted).

Twenty to thirty base profiles times four variants is 80 to 120 resumes. Score each variant several times, and you are running a few hundred calls. That is a cheap afternoon of compute and the single highest-value audit most teams never do.

This is the step most teams skip, and skipping it destroys the result. Large language models and scoring pipelines are not deterministic. The same resume scored twice can come back with different numbers. If you compare an elite-school score of 82 against an unknown-school score of 79 and declare a three-point gap meaningful, you may be reading noise.

The fix is to measure noise first. Take a handful of the base profiles, submit the identical text several times, and record the spread. If a profile usually scores around 80 and the runs land between 78 and 82, your noise band is roughly plus or minus two points. Only deltas that clear that band, consistently and in the same direction, count as signal.

The public safety precedent is useful. Turnitin's AI writing detector does not show a number below 20% because its own testing found a higher incidence of false positives in that range, and it flags those low results with an asterisk rather than a percentage (Turnitin). That is exactly the discipline a screener audit needs: a known uncertainty floor, declared in advance, below which you do not make claims.

A worked example makes the arithmetic concrete. Suppose five identical submissions of profile 7 score 78, 80, 81, 79, 80. Mean 79.6, spread 78 to 81, so your band is about plus or minus 1.5 around the mean. Round up and declare a two-point band. Any school variant whose mean sits within 79.6 plus or minus 2.0 is indistinguishable from a re-roll. A variant averaging 74 is outside the band and demands an explanation.

noise band, profile 7, five identical runs
runs:     78  80  81  79  80
mean:     79.6
range:    78 to 81
band:     +/- 2.0  (declare before testing)

variant means (five runs each), same profile
elite:    83.4   ->  +3.8  outside band
mid-tier: 80.2   ->  +0.6  inside band
unknown:  74.0   ->  -5.6  outside band
blank:    71.8   ->  -7.8  outside band

Read that table honestly. The mid-tier school is indistinguishable from noise. The elite school is higher than the profile's own baseline, and the unknown and blank conditions are lower. That pattern, spread across many profiles rather than one, is what school sensitivity looks like.

Now score all four variants, several times each, and compute a mean per variant per profile. The unit of analysis is the profile, not the individual run. You are looking for whether the ordering elite > mid-tier > unknown > blank holds across many profiles, and how large the gaps are relative to each profile's noise band.

A comparison table helps you decide what each outcome means.

Two interpretive cautions. First, school name can legitimately correlate with something the scorecard cares about, such as a specific program. If you cannot state in one sentence why a given school should change the score, the screener is not showing you a valid signal. Second, a single failing profile is an anecdote. Sensitivity is a pattern across your 20 to 30 profiles, so report the distribution, not the worst case.

It also helps to know what kind of screener you are testing. A keyword or ATS filter behaves differently from an LLM ranking prompt, and vendors that build their own screening tools concentrate this risk in-house. One analysis of build-versus-buy for recruiting AI puts it plainly: the prompt that scores perfectly on engineering resumes can behave differently on other candidate types, and without an audit trail nobody can explain why a specific applicant was screened out (Metaview). The planted-variation test is your audit trail for the school field specifically.

A one-time probe is useful. A probe you rerun whenever the prompt or model changes is a control. Treat the test set like a suite of unit tests.

1. Keep the profiles and variants in version control. Freeze the base profiles. When you change a prompt, rerun the whole suite and diff the variant means.
2. Record the noise band per profile, not globally. Some profiles will be noisier than others, and a global band hides that.
3. Log the score, the variant, and the run count. This gives you the evidence trail that a recruiter, a lawyer, or an auditor will ask for. Documenting the evidence behind each rating is what makes selection decisions defensible (Slite's hiring scorecard template; Pin).
4. Set a threshold and honor it. For example: if the elite-to-blank gap exceeds twice the noise band on more than a quarter of profiles, the screener fails the gate. Pick the rule before you see the data so you cannot rationalize the result after.
5. Re-run on every material change. New model, new prompt, new job description template, new scorecard weighting.

Retention matters here. The EEOC's recordkeeping guidance requires employers to keep employment records, including selection records, for defined periods, and a documented probe makes that obligation far easier to meet than a pile of unscored applications (Pin). Structured, evidence-backed scoring is not just fairer; it is easier to defend.

The probe answers a technical question: is the score sensitive to school name? The legal frame tells you why that matters. Under Title VII and the Uniform Guidelines, a selection procedure that disproportionately excludes a protected group and is not job-related and consistent with business necessity is exposed to a disparate impact claim, and no intent is required (BambooHR; OutSolve). School prestige is a weak proxy for ability and correlates with family income, geography, and other attributes you are not allowed to screen on. A screener that leans on it is building a disparity into the front of your funnel.

That is why the probe is worth more than a retrospective ratio for the school question specifically. It gives you a clean causal finding you can act on before any live candidate is affected. Fairness in AI recruitment is contested because different stakeholders mean different things by it, and a define-then-measure approach lets you state your standard and test against it rather than argue about it after the fact (Rigotti and Fosch-Villaronga, Computer Law & Security Review).

The decision rule at the end is simple. If variant means stay inside the noise band across your profile set, the screener passes the school gate and you proceed with monitoring. If they do not, you have three options: strip the school field before scoring and see whether the work history alone carries the score, add a written justification requirement so prestige has to be defended, or replace the scoring prompt with one anchored to the scorecard criteria. Test each fix with the same probe before trusting it.

When you want a second pair of eyes on the profile set or on how the screener is wired into your pipeline, a short working session beats another week of guesswork. You can book a free 30-minute AI discovery call to walk through your goals and get a tailored recommendation, no obligation.