Prevalence of Specification Sensitivity Across Meta-Analyses

confidence: high · 2 sources

What the evidence says

The sources on this page address one question: how often do analytic choices change the conclusion of a meta-analysis? The main source is a study of 358 behavioral science meta-analyses. It tests four outlier-handling treatments, each combined with several estimators. Two smaller claims come from a separate paper on Fisher's method for combining p-values. That paper shows a related instability in a different setting.

In the behavioral science sample, outlier handling usually moves the pooled effect very little. The median absolute change in Cohen's d is at most 0.047 and often much less. The conclusion about direction and size therefore rarely moves much for a typical meta-analysis.

Categorical conclusions change more often than the size of the effect. At least one treatment, combined with one estimator, reverses statistical significance in 11.5% of meta-analyses (41 of 358). It reverses the smallest-effect-of-interest assessment in 15.9% (57 of 358), using a threshold of |d| ≥ 0.20. The source also reports a sensitivity analysis with a lower threshold (c = 0.10). That analysis gives a 6.4% rate, counted over 715 meta-analysis-estimator combinations. The denominators differ: the first two rates count meta-analyses, and the 6.4% rate counts combinations.

The treatments differ in how often they change results. Winsorizing has the least effect. It changes specification sensitivity in 2.38% of 715 combinations, against 5.45% for DFBETAS. For the smallest-effect-of-interest assessment, the figures are 5.87% for Winsorizing and 7.12% for DFBETAS. The reversals are concentrated in results already close to the decision boundary. Strongly significant results essentially never change.

The Fisher's method paper shows the problem from the other side. Two sets of p-values can give the same combined statistic, T_F = 13.8 and p = 0.032, even though one has a single strong study and the other has three weak ones. In an empirical example, nearly identical p-values led the Adaptively Weighted Fisher method to assign different weights, (1,1,1) in one case and (1,0,1) in the other. Both examples are descriptive and come from one paper. They do not give a prevalence figure.

Where the evidence splits

No contradictions in the current sources.

Thresholds and dose-response

No threshold established in the current sources.

Open empirical questions

Do the reversal rates (11.5% for significance, 15.9% for the smallest effect of interest) hold in meta-analyses outside behavioral science?
How often does the choice of combining method, such as Fisher's method, change conclusions across a large sample of meta-analyses?
What mechanism explains why DFBETAS changes results more often than Winsorizing?
How do reversal rates change when outlier thresholds are set before results are seen?

Findings

Outlier handling treatments produce a small change in the pooled effect size of behavioral science meta-analyses

confidence — direction: highconfidence — magnitude: highconsensus: thin — one source

mechanism: outlier handling (four treatments) removes or down-weights extreme studies, which shifts the pooled estimate
effect: median absolute change in Cohen's d of at most 0.047, and often much less
conditions / moderators: measured on 358 behavioral science meta-analyses; generalization beyond them is not established

Tomas Havranek (2026), arXiv · doi:10.48550/arxiv.2607.23174v1 · https://arxiv.org/pdf/2607.23174v1 (retrieved 2026-10-06)

Outlier handling treatments reverse statistical significance in 11.5% of meta-analyses and the smallest-effect-of-interest assessment in 15.9%

confidence — direction: highconfidence — magnitude: highconsensus: thin — one source

mechanism: a small shift in the pooled estimate or its standard error moves a result across a decision cutoff
effect: significance reverses in 11.5% (41 of 358) of meta-analyses for at least one treatment-estimator combination. The smallest-effect-of-interest assessment (|d| ≥ 0.20) reverses in 15.9% (57 of 358). With a threshold of c = 0.10, the rate is 6.4% across 715 combinations.
conditions / moderators: reversals sit almost entirely in results already near the decision boundary; strongly significant results essentially never change. Validation outside behavioral science is not established.

Tomas Havranek (2026), arXiv · doi:10.48550/arxiv.2607.23174v1 · https://arxiv.org/pdf/2607.23174v1 (retrieved 2026-10-06)

Winsorizing produces less specification sensitivity than DFBETAS

confidence — direction: highconfidence — magnitude: highconsensus: thin — one source

mechanism: not established in the claims
effect: specification sensitivity changes in 2.38% of 715 meta-analysis-estimator combinations under Winsorizing and 5.45% under DFBETAS. The smallest-effect-of-interest assessment changes in 5.87% and 7.12%.
conditions / moderators: not established

Tomas Havranek (2026), arXiv · doi:10.48550/arxiv.2607.23174v1 · https://arxiv.org/pdf/2607.23174v1 (retrieved 2026-10-06)

Fisher's method gives identical results for different patterns of study-level p-values, and its adaptive weights are unstable

confidence — direction: highconfidence — magnitude: highconsensus: thin — one source

mechanism: Fisher's statistic sums log p-values, so different patterns of heterogeneity can give the same total. The adaptive weight estimate switches between classifications when inputs differ slightly.
effect: p-values (0.001, 1, 1) and (0.1, 0.1, 0.1) both give T_F = 13.8 and p = 0.032. In the empirical example, p-values (0.000391, 0.0962, 0.00211) gave weights (1,1,1) and p = 5.64 × 10−5. Values (0.000356, 0.1026, 0.00206) gave weights (1,0,1) and p = 5.22 × 10−5.
conditions / moderators: two illustrative examples only; prevalence is not measured

Zhiguang Huo (2017), arXiv · doi:10.48550/arxiv.1708.05084v1 · https://arxiv.org/pdf/1708.05084v1 (retrieved 2026-10-07)