How to Audit a Hiring Agent for Bias Without a Data Team

The conventional advice is a few hundred applications per stage per group. The source of that convention is less important than what it buys you: enough observations that a single unusual hire does not swing the ratio. If a group has 40 applicants and 4 hires, one hire moves the selection rate by 2.5 points. If it has 400 applicants and 40 hires, one hire moves it by a quarter of a point.

For a small team, exact random sampling is often unnecessary and sometimes impossible. The better approach is a census of a defined window. Pick the most recent complete quarter, or the most recent closed requisitions, and take every application that reached the stage you are auditing. That removes the sampling question entirely as long as the volume is workable. If it is not, sample systematically: sort applications by application ID and take every nth record, which is easy to explain and easy to repeat.

The uneven minimums are deliberate. Later stages have fewer people, and a small team cannot manufacture volume that does not exist. The honest move is to report a wide confidence interval, or simply to label a late-stage result as directional rather than conclusive. What you must not do is compute a four-fifths ratio on 8 people and treat it as a finding.

Two data hygiene rules protect the whole exercise. First, fix the window before you look at outcomes, otherwise it is too easy to extend a window until a disparity disappears. Second, record the date you pulled the data and the exact filters you applied, because the ATS will change underneath you and next quarter's numbers will not be comparable otherwise.

The arithmetic is simple enough to do on paper. The EEOC's four-fifths rule, as described in the BambooHR glossary entry on adverse impact, has four steps: compute each group's selection rate, identify the highest rate, divide each group's rate by the highest, and flag anything under 0.80. Berkshire Associates' four-step walkthrough spells out the same method, which has been used in discrimination analysis for decades.

Worked example. Suppose the resume screening stage for one role over one quarter looks like this:

• Group A: 300 applicants, 90 advanced. Selection rate 0.300.
• Group B: 250 applicants, 60 advanced. Selection rate 0.240.
• Group C: 120 applicants, 18 advanced. Selection rate 0.150.

The highest rate is Group A at 0.300. The impact ratios are 0.300 / 0.300 = 1.00 for Group A, 0.240 / 0.300 = 0.80 for Group B, and 0.150 / 0.300 = 0.50 for Group C. Group B sits exactly at the line, which the guidelines treat as generally not evidence of adverse impact, though it is close enough to watch. Group C is well below the line and is the finding that goes into the report.

In a spreadsheet, this is a handful of cells. Put applicants in one column, selected in the next, and a third column for the rate:

=IF(B2=0,"",B2/A2)

Call that column C. Then take the maximum of column C and divide every rate by it:

=IF(C2="","",C2/MAX($C$2:$C$5))

A pivot table gets you the counts. Never compute a rate on a zero denominator, and never let a spreadsheet silently treat a blank as a zero.

Once the basic ratio is in hand, a few refinements are worth the small effort. Compare ratios at every stage separately rather than only on the final hire rate, because a disparity at the screen can be washed out or created by later stages. Look at ratios by role family as well as overall, since one heavily technical role can dominate a mixed pool. And if you have the appetite, run a chi-square or Fisher's exact test; statistical significance is not the legal standard, but it stops you from chasing noise.

The 0.80 threshold is a screening device, not a verdict. The guidelines say a ratio below four-fifths will generally be regarded by enforcement agencies as evidence of adverse impact, and a ratio at or above it generally will not. That word "generally" carries a lot of weight. Two things follow.

First, a ratio below 0.80 does not mean your agent is biased. It means the stage deserves a closer look, exactly as the guidelines frame it: the four-fifths test is the first stage of analysis, and a closer examination looks at individual selection procedures. Pin's guide to adverse impact in hiring makes a related point that often surprises recruiters: disparities rarely show up at the offer stage, they show up at the top of the funnel, in screening and shortlisting. That is where an agent usually operates.

Second, a ratio above 0.80 does not clear you. With small samples the ratio is unstable, and with any sample it can mask a problem at one stage offset by a favorable result at another. The practical decision rule for a team without a data function:

When you do open a review, the question changes from "is there a disparity" to "is the procedure job-related and consistent with business necessity." That is the standard Title VII applies, as BambooHR's overview explains, and it is why a resume screen that rewards a specific degree or a specific job title can be defended or attacked on grounds completely separate from the model. In practice, most findings trace back to a criterion someone wrote into the prompt or the knockout rules, not to the language model's inner workings.

An audit that cannot be repeated is an opinion. The documentation should be short enough that a two-person HR team actually completes it, and complete enough that someone else could rerun it. A one-page template per audit cycle works:

• Scope: which roles, which requisitions, which date window, which decision points.
• Data source: ATS export name, filters applied, date pulled, who pulled it.
• Method: the formulas above, the groups compared, the minimum sample rule applied.
• Results: a table of counts, rates, and impact ratios per stage.
• Interpretation: which ratios fell below 0.80 and what you decided to do.
• Actions: owner, action, due date, and the follow-up audit date.

Two things make the record stronger than the numbers alone. First, keep the job-relatedness reasoning next to the ratio. If a stage shows a disparity but the criterion is a genuine requirement, say so in the same paragraph, with the evidence. Second, keep the version of the agent's configuration, meaning the prompt, the knockout rules, and the scoring rubric, with the audit. The Metaview discussion of building versus buying recruiting agents points out that when a recruiter asks why a specific applicant was screened out and nobody can answer because there is no audit trail, the tool is not production-ready. The same applies to the audit itself: if the configuration is not recorded, the finding cannot be tied to a cause.

Retention deserves a line in the template. The EEOC's recordkeeping requirements, summarized in Pin's scorecard guide, call for one year of retention generally and two years for larger federal contractors. Align your audit retention to the longer of that and your own state obligations, and note it.

Does the agent need to be explainable for this to work? No. The four-fifths analysis is an outcome measure. You are comparing selection rates, not inspecting reasoning. Explainability helps when you move to the review stage and want to know which criterion drove a rejection, but it is not a prerequisite for the audit.

What if the ATS does not store demographic data with the application? Many do not, by design. In that case you can still audit on proxies you lawfully hold, such as source channel or location where relevant, and you should prioritize getting voluntary self-identification data into an aggregate reportable form. A vendor or an internal engineer can usually produce the aggregate export. If you need an application built to pull it, Quanta takes a described plan and builds it with you.

How often should this run? Quarterly for the stages that touch the most candidates, and after any change to the agent's prompt, rubric, or model. A configuration change is the single most common reason a clean audit goes bad.

What does a defensible audit look like if a claim arrives? It shows a fixed method, a defined window, a recorded data source, results at each stage, and a documented response to anything below 0.80. That is a process that was monitored and corrected, which is a materially different posture from no record at all.

Who should do the work? Someone outside the hiring decision, even if that person is a generalist. The separation between computing the numbers and making the offer is the control that keeps the audit honest.