Gender-Bias Evaluation Experiments Outside Veterinary Medicine and in Actual Hiring Decisions

What the evidence says

The strongest causal evidence on this page comes from a single natural experiment far outside NLP or medicine: the adoption of blind (screened) auditions in symphony orchestras. Using variation in when orchestras adopted screens, Goldin (2000) finds that blind auditions increase a woman's probability of advancing from the preliminary round by 11 percentage points when no semifinal round follows, and increase her probability of winning the final round by 33 percentage points. These estimates come from a small subset of musicians who auditioned both with and without a screen (639 person-rounds for the preliminary estimate, 67 for the final-round estimate), and the paper notes some estimates carry large standard errors. At the level of overall hiring outcomes, blind auditions account for about one-third of the increase in the proportion of women among new hires, and about 25% of the increase in the proportion of orchestra musicians who are female — the latter figure from a less rigorous roster-based (associational) analysis rather than the audition fixed-effects model.

Outside that domain, a separate line of evidence looks at gender bias in machine translation. Cho (2019) finds that translation systems differ in how well they preserve gender-neutral pronouns: Google Translator scored 0.2992 versus Kakao Translator's 0.1184 on the same test corpus, and Naver Papago showed an unusual pattern of translating researcher and engineer into female pronouns more often than the other two systems, which the source describes as differing from expected social-prejudice patterns. Within the same system, formal Korean pronouns were translated with stronger male bias than informal pronouns, and occupation words showed more gender bias than sentiment or adjective words. Politeness of sentence structure showed no measured effect on translation gender bias.

A third, methodological line of evidence (Orgad 2022) does not measure gender bias in any real-world system directly; instead it measures how NLP researchers measure it. Most gender-bias studies in this literature use only intrinsic metrics, which the source states makes it difficult to determine what harm the underlying bias actually causes. Consistent with that gap, most papers measuring downstream (extrinsic) metrics measure three or fewer of them despite more being possible. This is a methodological-quality finding, not a bias-magnitude finding — it describes what the field measures, not how biased any given model is.

On medical students specifically, the evidence is thin and largely descriptive. One cross-sectional survey (anon 2025) reports that 64.35% of male medical students believed condoms reduce sexual pleasure, versus 87.85% of female students who expressed concern about oral contraceptive side effects — these are different outcome measures for each gender, not a comparison of the same belief across genders. The same source reports that gender differences in attitudes toward abortion "were evident," particularly regarding non-medical-context abortion and contraceptive-method perceptions, but supplies no quantification for that specific claim.

The topic name asks about gender-bias evaluation "in actual hi[ring]" — the orchestra-audition evidence is the only source here that measures a real hiring outcome rather than a translation output or a meta-level count of what researchers measure. No source on this page evaluates gender bias in hiring outside music performance, and no source connects the machine-translation or intrinsic/extrinsic-metric findings to any real-world harm or decision.

Where the evidence splits

Orgad (2022) reports that intrinsic and extrinsic bias metrics can move independently or even reverse relative to each other, but this is not a disagreement between two studies on the same quantity — it is one methodological source's own set of demonstrations (WEAT not correlating with extrinsic metrics per Goldfarb-Tarrant et al. 2021; a Pearson correlation for precision reversing sign between balanced and imbalanced test sets on the same model and dataset; compression's correlation with extrinsic metrics dropping from r²=0.567 to r²=0.025 across datasets). These are presented as evidence that intrinsic and extrinsic metrics measure different things, not as a contested empirical question with two camps.

Within the orchestra-audition evidence, blind auditions increase advancement in the preliminary round (+11 points) and the final round (+33 points), but have a negative effect on advancement from the semifinal round. Goldin (2000) calls this semifinal-round result "anomalous" and suggests it may reflect committees behaving more cautiously toward women once anonymity is removed, rather than straightforward bias — but the source itself frames this as an unresolved interpretive uncertainty, not a settled explanation. The magnitude of the semifinal-round effect is not specified.

Thresholds and dose-response

No threshold established in the current sources.

Open empirical questions

• No source here measures gender bias in actual hiring decisions outside symphony orchestras — whether blind-evaluation effects generalize to other hiring contexts is unmeasured.
• No source connects intrinsic or extrinsic NLP bias metrics to a measured real-world harm or downstream decision outcome.
• Whether the semifinal-round reversal in orchestra auditions reflects evaluator caution, a different mechanism, or measurement noise is not established by the source.
• Whether the machine-translation gender-bias patterns found for Korean-English translation generalize to other language pairs or systems is not measured.