NYC Local Law 144 Bias Audit Results

confidence: high · 2 sources

What the evidence says

Local Law 144 requires bias audits of automated hiring tools, and the sources here describe what those audits contain and what they leave out. The evidence is descriptive. It covers the text of the law and a set of published reports. It does not measure whether audits change hiring outcomes.

The law sets the audit's scope narrowly. It requires one fairness metric, the ratio of selection rates between protected categories. It also lets an independent auditor exclude any category that is less than 2% of the audit data from the required calculations. One source reads this as systematically excluding American Indian, Alaska Native, Native Hawaiian, Pacific Islander, Two or more race, and Some Other Races groups, along with intersectional groups. Under the law's design, a group's small share of the data removes it from the required calculation.

The law also leaves the audit's inputs open. It does not specify data requirements, so an auditor may use any dataset, from anywhere, for any time range. The authors saw population breakdowns in published audits that lead them to believe the data may not be from NYC. That is an inference from breakdowns, not a count of audits using non-NYC data.

Published results show ratios above the 0.8 benchmark used in federal employment discrimination law in nearly all cases. Of 386 measurements, nine fell below 0.8, and 96% were above it. The sources give two reasons to read that figure with care. The law relies on the 80/20 rule but does not enforce it, so an audit can find a model outside the range with no action required. And the study authors saw one employer that may have withheld an audit report. The archived copy of that report shows ratios below 0.8 in two demographic areas. The authors cannot confirm why it was unpublished. The published set may therefore leave out the least favorable results.

The reports are also hard for job seekers to use. In the study's qualitative review of 13 unique reports, none included enough explanation to inform a job seeker or guide a decision to opt out of applying. Reports were mostly PDF data tables of scoring rate, selection rate, and ratio by sex and race/ethnicity, with minimal explanation. That judgment came from the authors and trained student investigators. It is not a quantitative measure of comprehension.

Nothing on this page measures downstream outcomes, such as changes in who is hired, whether employers change tools after an audit, or how many covered employers publish at all.

Where the evidence splits

No contradictions in the current sources.

Thresholds and dose-response

Two cutoffs appear in the sources, both set by the law rather than found in outcome data.

Categories below 2% of the audit data may be excluded from the required calculations.
A ratio below 0.8 (the 80/20 range) marks performance outside the acceptable range, but the law does not enforce it. An audit can find a model outside this range and no action follows.

No source shows an outcome change at either cutoff. Otherwise, no threshold is established in the current sources.

Open empirical questions

How many audits use the 2% exclusion, and which groups are dropped from the required calculations?
What share of covered employers publish their audit reports, and how many reports are withheld?
How many audits use data from outside NYC, and does that change the ratios reported?
Do audits change employers' tools or hiring outcomes, and do job seekers act on the published reports?

Findings

Local Law 144 decreases the coverage of small demographic groups in audit calculations.

confidence — direction: highconfidence — magnitude: highconsensus: thin — one source, resting on the explicit text of the law.

mechanism: The law lets an independent auditor exclude any category under 2% of the audit data from the required ratio calculations. Small groups and their intersections can therefore be left out.
effect: Groups named in the source (American Indian, Alaska Native, Native Hawaiian, Pacific Islander, Two or more race, Some Other Races, and intersectional groups) can be excluded. No count of audits that used the exclusion is given.
conditions / moderators: Applies where a category is under 2% of the data used for the audit. The exclusion is permitted, not required.

Gemma Galdon Clavell (2024), arXiv · doi:10.48550/arxiv.2501.10371v1 · https://arxiv.org/pdf/2501.10371v1 (retrieved 2026-10-01)

The law requires one metric and no enforcement action, which leaves audit results without a required consequence.

confidence — direction: highconfidence — magnitude: highconsensus: thin — one source, leaning toward a narrow requirement with no enforcement.

mechanism: The law requires only the selection-rate ratio between protected categories. It relies on the 80/20 rule but does not enforce it.
effect: An audit can find a model performing outside the 80/20 range and no action would be taken. No effect size applies.
conditions / moderators: Based on the law's text and observed audit practice. Other fairness metrics are not required.

Gemma Galdon Clavell (2024), arXiv · doi:10.48550/arxiv.2501.10371v1 · https://arxiv.org/pdf/2501.10371v1 (retrieved 2026-10-01)

The law's silence on data requirements produces audits built on datasets of varying origin and period.

confidence — direction: highconfidence — magnitude: mediumconsensus: thin — one source.

mechanism: The law does not specify data requirements, so auditors may use any dataset, from anywhere, for any time range.
effect: Audit data can differ in geography and time period. The authors believe some published audits use non-NYC data, based on population breakdowns. No count is given.
conditions / moderators: Based on observed population breakdowns, not confirmed data provenance.

Gemma Galdon Clavell (2024), arXiv · doi:10.48550/arxiv.2501.10371v1 · https://arxiv.org/pdf/2501.10371v1 (retrieved 2026-10-01)

Published audit reports show ratios above 0.8 in nearly all measurements, with possible selective publication.

confidence — direction: highconfidence — magnitude: high for the published count, medium for the withheld reportconsensus: thin — one descriptive study.

mechanism: Employers post audit reports. The study found at least one employer that may have withheld a report with ratios below 0.8. Selective posting could raise the share of favorable results among published reports.
effect: Of 386 measurements, nine were below 0.8 and 96% were above it. One unpublished, archived report showed ratios below 0.8 in two demographic areas.
conditions / moderators: Covers published reports only. The authors cannot confirm why the report was unpublished. They speculate that legal liability concerns played a role.

Lucas Wright (2024) · doi:10.1145/3630106.3658998 · https://doi.org/10.1145/3630106.3658998 (retrieved 2026-10-01)

Published audit reports include too little explanation for job seekers to make informed decisions.

confidence — direction: highconfidence — magnitude: mediumconsensus: thin — one qualitative study.

mechanism: Reports are mostly PDF data tables of scoring rate, selection rate, and ratio by sex and race/ethnicity, with minimal explanation.
effect: None of the 13 unique reports reviewed included enough explanation to inform job seekers or guide a decision to opt out. No measured effect on job seeker behavior.
conditions / moderators: Rests on the judgment of trained student investigators and the authors. There is no quantitative comprehension measure.

Lucas Wright (2024) · doi:10.1145/3630106.3658998 · https://doi.org/10.1145/3630106.3658998 (retrieved 2026-10-01)