Sensitivity of Published Effect Sizes to Sample-Restriction Rules

confidence: high · 4 sources

⚠
Working research — not a recommendation.

This page records what the evidence says a policy does, not what anyone should do about it. It has been synthesized from the cited sources but has not been independently reviewed.

What the evidence says

The claims on this page do not measure sample-restriction rules such as inclusion criteria or exclusion cutoffs. They cover three related kinds of restriction: how many observations a study draws, which results get reported, and which analysis specifications get run. In each case the published estimate depends on the restriction rule.

Sample size changes the estimate in a simulation of debris flow susceptibility models. Across 1000 random samples at sizes between n = 50 and n = 5000, predicted probabilities for debris flow release differed, on average, by 7 to 23 percentage points, depending on the study area and sample size. Model diversity decreases as sample size grows, then reaches a local minimum or a plateau. Larger samples do not reduce it further, and the maximum sample size is bounded by the autocorrelation range of the spatial data. This evidence comes from one study of one hazard type.

Restricting reports to results that exceed the α = 0.05 threshold inflates reported effect sizes. Szűcs states that papers mostly report exaggerated effect sizes because typically only results above that threshold are reported. The inflation is larger when statistical power is lower. Published false positives, where the null is true but p ≤ α, also overestimate the size of non-existent or tiny effects. The mechanism is selection on the test outcome. The sources do not quantify the size of the inflation.

Costello adds two related points. The probability of passing the standard test increases with sample size whether or not a true effect exists. Standard testing also ignores between-experiment variation. When that variation is accounted for in Many Labs 1 data, around 10% of results with point-form p < 0.00001 had psig > 0.05. The source also notes that many such results in psychology and related fields fail to replicate, but it gives no rate.

Clemens cites two points about analyst flexibility. Casey et al. (2012) show they could have obtained nearly any result with plausible changes to their regressions, had a pre-analysis plan not constrained them. Any result can also be turned into a null by running modified tests that are underpowered by construction. These are cited principles, and the sources here do not test them across domains.

Where the evidence splits

No contradictions in the current sources.

Thresholds and dose-response

No threshold established in the current sources.

Open empirical questions

Findings

Smaller or differently drawn samples change predicted debris flow probabilities by 7 to 23 percentage points

confidence — direction: highconfidence — magnitude: highconsensus: thin (single simulation study)
mechanism
random subsamples of different sizes produce different fitted models, which give different predicted probabilities
effect
predicted probabilities differed on average by 7 to 23 percentage points, depending on study area and sample size (1000 random samples per size, n = 50 to n = 5000)
conditions / moderators
varies by study area and sample size; whether the range holds across other debris flow systems and estimation methods is not established

Larger samples decrease model diversity until it plateaus

confidence — direction: highconfidence — magnitude: mediumconsensus: thin
mechanism
more observations constrain the fitted models, but the usable sample size is capped by the autocorrelation range of the spatial data
effect
model diversity decreases with sample size and reaches a local minimum or plateau; larger samples do not reduce it further (no numeric size given)
conditions / moderators
whether the plateau is specific to the study areas or a general property of the geofactors is not established

Reporting only results above α = 0.05 increases reported effect sizes, more so at low power

confidence — direction: highconfidence — magnitude: mediumconsensus: settled on direction (the sources describe a mechanical consequence of selection); magnitude not established
mechanism
selection on the test outcome keeps only the larger estimates; this includes false positives where the null is true but p ≤ α, which overestimate non-existent or tiny effects
effect
direction is an increase in reported effect size relative to the true effect; the lower the power, the larger the overstatement; no magnitude is quantified
conditions / moderators
magnitude depends on base rates and the strength of publication selection, which are not quantified here

Standard significance testing increases the chance of passing with sample size and misses replication failure

confidence — direction: highconfidence — magnitude: high for the 10% figure in that dataset; low for general replication ratesconsensus: thin (one author's analysis, plus cited replication projects)
mechanism
the standard test counts only within-experiment random variation and ignores between-experiment variation, so total random variation is underestimated
effect
the probability of a result passing the test increases with sample size, whether or not a true effect exists; with between-experiment variation included, around 10% of Many Labs 1 results with p < 0.00001 had psig > 0.05
conditions / moderators
the 10% figure comes from one replication dataset (Many Labs 1); the source cites many non-replications in psychology and related fields but gives no rate

Changing the regression specification or the power of a test changes the result

confidence — direction: highconfidence — magnitude: mediumconsensus: thin (cited principle and one cited example)
mechanism
analyst choices over specification and test power are not constrained unless fixed in advance, for example by a pre-analysis plan
effect
nearly any result can be obtained with plausible specification changes; any result can be turned into a null by underpowered modified tests (no numeric size given)
conditions / moderators
a pre-analysis plan constrained the analysis in the Casey et al. case; whether this holds across other contexts, datasets, and effect sizes is not established