Inclusion and Exclusion Criteria Effects on Published Effect Sizes

confidence: high · 4 sources

What the evidence says

The sources on this page show that choices about which studies, patients, or observations to include can change a reported effect size. They can change its direction too. The best-quantified case is a set of 9,216 meta-analysis scenarios comparing nalmefene and naltrexone. Varying inclusion and exclusion criteria and analytical models produced effect sizes from −0.37 to 0.31, with a median of −0.04. The 1st percentile was −0.29, favoring nalmefene. The 99th percentile was +0.29, favoring naltrexone. The authors call this a Janus effect: the same evidence yields statistically significant results in both directions.

Statistical significance also depended on the direction of the result. Of the meta-analyses with a negative effect size, 7.1% (425/5961) were statistically significant at p < 0.05. Of those with a positive effect size, 18.9% (616/3255) were. The sources do not explain why the two directions differ. A separate review of respiratory trials says that excluding studies based on outcome measure "may potentially have altered" the effects in meta-analyses. That claim is hedged and carries no quantified before-and-after comparison, so it is low-confidence.

Within a single trial, the selection threshold itself changes the observed treatment effect. The mechanism is what the source calls the SICE effect. It can arise when treatment changes the correlation between a selection marker and the response of interest. A mathematical result in the source shows that the effect grows as the selection threshold rises. In an insomnia trial, the observed drug effect on latency to persistent sleep (LPS) was 39% larger than in the unselected population at a 15-minute threshold. At a 20-minute threshold it was 106% larger. Simulations also show that in larger trials the SICE effect is more likely to reach statistical significance, even when its size is small.

A critique of state-level panel studies on execution and homicide shows the same sensitivity through analytic choices instead of inclusion rules. The significance of coefficients on commutations and removals depends on which lag structure is used. A joint test across all six lags gave F = 1.09 (P = 0.38). Restricting to lags four to six gave F = 2.62 (P = 0.06). The author also reports that state and time fixed effects account for 89% of the variation in homicide rates. The remaining covariates account for less than 2%, and sanction risk variables for less than 0.5%. The author says a few influential observations produce "overwhelming model uncertainty." The sources here document these patterns in three domains. They do not measure how common the problem is across fields.

Where the evidence splits

No contradictions in the current sources.

Thresholds and dose-response

In the insomnia trial, the observed drug effect rose with the LPS inclusion threshold. It was 39% larger than in the unselected population at 15 minutes and 106% larger at 20 minutes. The source also shows mathematically that the SICE effect increases monotonically with a stricter threshold. The source states no cutoff at which the effect appears or reverses. Beyond these two thresholds, no threshold is established in the current sources.

Open empirical questions

How often do inclusion and exclusion choices flip the direction of pooled effects in fields other than the nalmefene and naltrexone comparison?
Why are positive-effect meta-analyses more often statistically significant than negative-effect ones in the 9,216-scenario set?
How large is the change in pooled effects when studies are excluded by outcome measure, as in the respiratory meta-analyses?
Does the SICE effect appear in trials whose selection markers differ from baseline sleep latency, and at what magnitude?

Findings

Varying inclusion and exclusion criteria and analytical models produces effect sizes in opposite directions from the same evidence.

confidence — direction: highconfidence — magnitude: highconsensus: thin — one meta-analysis, but with explicit quantification

mechanism: different combinations of included studies and analytical models yield different pooled estimates from the same underlying literature
effect: effect sizes ranged from −0.37 to 0.31 across 9,216 scenarios, with a median of −0.04; the 1st percentile was −0.29 (favoring nalmefene) and the 99th was +0.29 (favoring naltrexone)
conditions / moderators: shown in nalmefene versus naltrexone meta-analyses; not established for other topics

Clément Palpacuer (2019), BMC Medicine · doi:10.1186/s12916-019-1409-3 · https://bmcmedicine.biomedcentral.com/track/pdf/10.1186/s12916-019-1409-3 (retrieved 2026-10-06)

Meta-analyses with positive effect sizes were more often statistically significant than those with negative effect sizes.

confidence — direction: highconfidence — magnitude: highconsensus: thin — one source

mechanism: not established in the source
effect: 7.1% (425/5961) of negative-effect meta-analyses and 18.9% (616/3255) of positive-effect meta-analyses were significant at p < 0.05; the denominator is the meta-analyses in each direction group
conditions / moderators: same nalmefene versus naltrexone scenario set; not established elsewhere

Clément Palpacuer (2019), BMC Medicine · doi:10.1186/s12916-019-1409-3 · https://bmcmedicine.biomedcentral.com/track/pdf/10.1186/s12916-019-1409-3 (retrieved 2026-10-06)

Excluding studies by outcome measure may alter meta-analytic effect sizes.

confidence — direction: lowconfidence — magnitude: lowconsensus: thin — a hedged statement with no quantification

mechanism: excluded studies may differ in outcome measure from included ones, so the pooled estimate changes
effect: direction and magnitude not specified
conditions / moderators: not established

Reeves J (2026), ERJ open research · doi:10.1183/23120541.00550-2026 · https://europepmc.org/articles/PMC13591289?pdf=render (retrieved 2026-10-06)

Stricter patient-selection thresholds increase the observed treatment effect in a trial.

confidence — direction: highconfidence — magnitude: mediumconsensus: thin — one source, combining a mathematical result, simulation, and one real trial

mechanism: the SICE effect: when treatment changes the correlation between the selection marker and the response, selecting on the marker shifts the observed treatment difference; a larger threshold gives a more prominent effect
effect: in an insomnia trial, the observed effect on LPS was 39% larger than the unselected scenario at a 15-minute threshold and 106% larger at 20 minutes
conditions / moderators: requires treatment to alter the marker-response correlation; with larger trials, the effect is more likely to be statistically significant even when small

Junshui Ma (2020), arXiv · doi:10.48550/arxiv.2001.02036v1 · https://arxiv.org/pdf/2001.02036v1 (retrieved 2026-10-06)

Conclusions in state-level panel studies of execution and homicide depend on lag structure, variable operationalization, and a few influential observations.

confidence — direction: highconfidence — magnitude: highconsensus: thin — one critique paper

mechanism: fixed effects absorb most of the variation, leaving little for the causal covariates, so coefficients depend on specification and on a few observations
effect: the joint F-test on execution-rate lags gave P = 0.38 with six lags and P = 0.06 with lags four to six; fixed effects account for 89% of the variation in homicide rates, other covariates for less than 2%, and sanction risk variables for less than 0.5%
conditions / moderators: state-level panel regressions on this question; the source reports high sensitivity to how the Republican vote share instrument is operationalized, with no effect size given

Aaron J. Chalfin (2012), Journal of Quantitative Criminology · doi:10.1007/s10940-012-9168-8 · https://link.springer.com/content/pdf/10.1007/s10940-012-9168-8.pdf (retrieved 2026-10-06)