Outcome Audits of Deployed AI Hiring Systems Using Actual Protected Demographic Data

confidence: high · 3 sources

⚠
Working research — not a recommendation.

This page records what the evidence says a policy does, not what anyone should do about it. It has been synthesized from the cited sources but has not been independently reviewed.

What the evidence says

The sources here are thin on the exact question, which is outcome audits of deployed AI hiring systems that use actual protected demographic data. Almost all of it is secondhand: two review papers summarizing other studies and one audit of personality-scoring tools. None of the claims reports a full audit of real hiring outcomes broken out by protected group, and none gives a selection-rate ratio.

The clearest case is Amazon's recruiting algorithm, described in both reviews. One review says it ran in test mode. It assigned lower scores to women's resumes when ranking candidates. The other review says the system penalized resumes that mentioned women's colleges or used verbs more frequently used by female candidates. Both name the same mechanism. The algorithm was trained on data from current top performers, who were mostly male, and it learned to treat features associated with women as predictors of lower hiring likelihood. The claims give the direction but no score gap.

Two other findings concern tools at other stages of hiring. Facebook ads for supermarket cashier positions were shown to an audience that was 85% women, which suggests that sourcing algorithms can produce group disparities before any applicant is screened. Facial recognition software shows disparities in error rates across gender and race, according to three cited studies. The claim gives no rates. A separate line of work finds that commercial speech recognition systems from Apple, IBM, Google, Amazon, and Microsoft have higher word error rates for Black speakers than for White speakers on the same spoken content. These are not hiring audits, but they bear on any hiring tool that transcribes or analyzes speech.

The one audit with protected-group data tested personality scores on resumes submitted in different formats. The full sample stayed above a rank-order stability of 0.90 for Steadiness and Conscientiousness. The female subgroup (N = 33) and the English-primary-language subgroup (N = 56) fell below 0.90. These subgroups are small, and the claim does not say whether the gap is statistically meaningful.

One finding concerns human raters, not an algorithm. Recruiter ratings of applicants' Facebook information favored female and Caucasian applicants and were unrelated to later job performance. It is a useful comparison point, but it does not audit an AI system.

On regulation, New York City Local Law 144 requires third-party bias audits and public disclosure of the resulting ratios before automated employment decision tools are deployed. The EU AI Act classifies AI used in labor-related decision-making as high risk. The sources describe these rules but report no audit results produced under them.

Where the evidence splits

No contradictions in the current sources.

Thresholds and dose-response

No threshold established in the current sources.

Open empirical questions

Findings

AI resume-screening models trained on historical hiring data decrease resume scores for women relative to men

confidence — direction: highconfidence — magnitude: lowconsensus: thin — a single case, consistent across both descriptions
mechanism
the model is trained on data from current top performers, who were mostly male, and learns to associate female-coded features with lower hiring likelihood
effect
decreases scores for women's resumes in candidate ranking; penalizes mentions of women's colleges and female-associated verbs such as "volunteered" and "mentored"; no score gap or baseline is given
conditions / moderators
one company's system, described in test mode by one review and as an internal system used 2014–2017 by the other; both reviews rely on the same press report, so this is one case

Targeted job advertising algorithms produce a skewed audience by gender

confidence — direction: highconfidence — magnitude: highconsensus: thin — one reported audit
mechanism
the ad-delivery algorithm selects who sees a job ad, so disparities arise at the sourcing stage, before any application
effect
85% of the audience for supermarket cashier ads on Facebook was women
conditions / moderators
one occupation and one platform; no comparison audience or baseline is given

Facial recognition and speech recognition components show higher error rates for some demographic groups

confidence — direction: highconfidence — magnitude: mediumconsensus: settled for the existence of the disparities; thin for their effect on hiring decisions
mechanism
not established in the claims beyond performance differing by group
effect
facial recognition error rates differ across gender and race, with no rates given; commercial speech recognition has higher word error rates for Black speakers than for White speakers on the same spoken content
conditions / moderators
neither finding is an audit of hiring outcomes; the speech result covers systems from Apple, IBM, Google, Amazon, and Microsoft

Personality-scoring tools give less stable rankings for women and English-primary speakers when the resume format changes

confidence — direction: highconfidence — magnitude: mediumconsensus: thin — one audit
mechanism
not established; the audit perturbed the input format and compared rank-order stability
effect
Steadiness and Conscientiousness fall below r = 0.90 for female (N = 33) and English-primary (N = 56) subgroups, while the full sample does not
conditions / moderators
small subgroups; the claim gives no full-sample correlations and no test of whether the gap is meaningful

Recruiter ratings of applicants' social media information favor female and Caucasian applicants and show no association with job performance

confidence — direction: highconfidence — magnitude: highconsensus: thin — one study
mechanism
not established
effect
subgroup differences favoring female and Caucasian applicants; ratings unrelated to later job performance; no magnitudes given
conditions / moderators
human raters, not an AI system