Quantified Effect Size of Algorithmic/AI Hiring-Tool Bias versus Human Recruiters

confidence: high · 1 sources

⚠
Working research — not a recommendation.

This page records what the evidence says a policy does, not what anyone should do about it. It has been synthesized from the cited sources but has not been independently reviewed.

What the evidence says

The available evidence comes from one large-sample study of a recruiting platform's candidate-shortlisting data (58,765 jobs over 27 months), comparing gender fairness across AI-only, human-only, and hybrid human+AI selection, measured with a CDP score. On its own, AI-driven candidate recommendation produces lower gender fairness than human recruiters' manual selection: AI Recommended Top-K CDP is 0.699 versus 0.813 for Human Contacted, a 0.114-point gap (p<0.001), with high confidence on both direction and magnitude given the sample size and significance testing.

Combining human and AI recruiting increases fairness beyond either channel alone. Human+AI Contacted CDP reaches 0.854, a 0.041-point increase over human-only recruiting (0.813) and a 0.155-point increase over AI-only recruiting (0.699), with statistically significant improvements across the 27-month series (p<.05 or lower) and a consistent pattern across candidate subsets (viewed, clicked, contacted). The claims describe this as more than an additive effect: the combined score exceeds what summing the separate human and AI contributions would predict, which the source calls a synergistic interaction. The mechanism for that synergy is not established — the source states directly that "we were not able to qualify exactly how this interaction takes place."

Human review of AI output increases fairness relative to AI recommendations alone, but the reported magnitude of that increase varies by which comparison is used. One claim reports candidates found through manual search after viewing AI recommendations reach a CDP of 0.876, compared to 0.791 for candidates contacted from the AI recommendation list (ΔCDP = 0.085, p<0.001) and 0.813 for candidates contacted through manual search alone (ΔCDP = 0.063, p<0.01). A separate claim reports AI-oversight CDP in the 0.791–0.793 range against an AI-only Recommended baseline of 0.642, an improvement of roughly 0.15 points that still falls short of the human-only 0.813 figure. Both describe the same qualitative pattern — human oversight of AI recommendations increases fairness but does not eliminate the shortfall relative to human-only recruiting — at medium confidence on magnitude, since the source notes the interaction mechanism was not directly tested.

The evidence is limited to a single outcome measure (CDP, a candidate-selection fairness score) and a single protected characteristic (gender). No claim here addresses whether these shortlisting-fairness gaps translate into differences in actual hiring or offer rates, and no claim addresses bias by race, age, disability, or other protected characteristics. All findings originate from one study and one platform's dataset, so there is no independent corroboration to check them against.

Where the evidence splits

The claims report two different CDP values for the AI-only "Recommended" baseline: 0.699 (used in the AI-vs-human comparison and the hybrid-synergy comparison) and 0.642 (used in the human-oversight-mitigation comparison). Both are attributed to the same source and metric, but the claims do not explain the discrepancy — it is unexplained by the sources on this page.

Separately, one claim's headline effect size (Post-AI oversight Contacted CDP 0.876 vs AI-only Recommended Top-K CDP 0.699, a 0.177-point difference) does not match the comparison actually stated in its own verbatim quote, which instead reports 0.876 against 0.791 (ΔCDP = 0.085, p<0.001) and against 0.813 (ΔCDP = 0.063, p<0.01) — figures that do not involve 0.699 at all. The direction of the finding (manual search after AI exposure outperforms AI-only or human-only alone) holds under either version, but the two stated magnitudes for the same comparison are inconsistent, and no claim here explains why.

Thresholds and dose-response

No threshold established in the current sources.

Open empirical questions

Findings

AI-only candidate recommendation decreases gender fairness relative to human-only manual selection

confidence — direction: highconfidence — magnitude: highconsensus: thin — single study, but internally strong (large sample, significance testing); direction not contradicted elsewhere in these sources
mechanism
not established beyond "relying on fully-automated candidate recommendation risks amplifying gender unfairness" — no specific algorithmic pathway is described
effect
AI Recommended Top-K CDP 0.699 vs Human Contacted CDP 0.813, a 0.114-point decrease for AI (p<0.001)
conditions / moderators
not established

Hybrid human-plus-AI recruiting increases gender fairness beyond either human-only or AI-only recruiting, more than additively

confidence — direction: highconfidence — magnitude: high for the combined-vs-separate comparison; medium for the claim that the effect is specifically synergistic (mechanism unverified)consensus: thin — single study
mechanism
described as a synergistic interaction between human and AI selection stages; the source states the specific mechanism was not characterized ("we were not able to qualify exactly how this interaction takes place")
effect
Human+AI Contacted CDP 0.854 vs Human-only 0.813 (+0.041) and vs AI-only 0.699 (+0.155); combined effect exceeds the sum of the separate human and AI contributions
conditions / moderators
consistent across viewed, clicked, and contacted candidate subsets over a 27-month period; not tested outside this platform/dataset

Manual search conducted after exposure to AI recommendations increases gender fairness relative to AI recommendations or manual search alone

confidence — direction: highconfidence — magnitude: high, though the two reported magnitudes for this comparison are inconsistent with each other (see split above)consensus: thin — single study
mechanism
not established beyond the observed sequencing (AI exposure followed by manual search)
effect
Contacted-via-manual-search-after-AI CDP 0.876 vs Contacted-from-AI-recommendation-list CDP 0.791 (ΔCDP = 0.085, p<0.001) and vs Contacted-via-manual-search-alone CDP 0.813 (ΔCDP = 0.063, p<0.01); a separate summary in the same claim states 0.876 vs an AI-only Recommended Top-K CDP of 0.699 (0.177-point difference) — see split above
conditions / moderators
not established

Human oversight of AI recommendations increases fairness relative to AI-only recommendations but does not eliminate the gap to human-only recruiting

confidence — direction: mediumconfidence — magnitude: mediumconsensus: thin — single study; magnitude also affected by the AI-only baseline discrepancy noted above
mechanism
not directly tested; source notes it could not qualify how the human-AI interaction takes place
effect
AI-oversight Clicked/Contacted CDP 0.791–0.793 vs AI-only Recommended CDP 0.642, an increase of roughly 0.15 points, remaining below Human-only CDP 0.813
conditions / moderators
not established