Selection-Induced Effect Inflation in Clinical Trials

confidence: high · 4 sources

What the evidence says

Two kinds of selection can inflate what a trial appears to show. One is selection of who enters each arm, and the other is selection of which patients, doses or trials count toward the result. The sources here cover both, and all of them rest on simulations, theory or one real trial. None is a field study of trials in general.

Selection bias in allocation increases the type I error rate under every randomization procedure tested. In simulations, even completely randomized design (CRD) reached about 7%, against a nominal 5%. The author reports that this is not a numerical artifact. Extra simulations found that inflation depends on the size of the bias effect but not on sample size. The procedure matters a great deal. Permuted block design with block size 2, PBD(2), reached a type I error of 38% to 39%. The big stick design BSD(3) reached 8% to 9% with the same three tests, and the source puts the range for MTI and biased coin designs at about 6% to 9%. The author's summary is that the more random the design, the less susceptible it is to selection bias. A theoretical literature cited in the same paper reaches the same direction for small block sizes, but it gives no magnitude.

Selecting patients on a baseline marker can also inflate the observed treatment effect. The source calls this the selection-induced contrast effect (SICE). The mechanism is that treatment changes the correlation between the selection marker and the outcome differently than control does. A stricter selection threshold increases the inflation. In a real insomnia drug trial, the observed effect on Latency to Persistent Sleep was 39% larger than in the unselected population at a 15-minute threshold. It was 106% larger at a 20-minute threshold. In a parametric simulation with a true drug effect of zero, selecting patients with a marker value of at least 0.6 raised the probability of a claimed drug effect from the nominal 5% to about 98%. The size of the inflation does not depend on sample size. The variance of its estimate falls as patients are added, so a large enough trial will eventually detect a non-zero inflation. The sources do not say how often these conditions arise in real trials. The effect depends on covariance differences between arms, which the source notes are often unobservable before the trial ends.

In Phase I dose-finding, one source shows a related order-dependence problem. The "winner-takes-all" dose assignment rule in long-memory designs gives early cohorts a disproportionately large influence. Run-to-run variability in the number of cohorts allocated to the true dose follows the order in which participants enter. When the same thresholds were rerun with different orders, the variability pattern was similar. The author also reports little evidence that early settling on a dose goes with better maximum-tolerated-dose selection, and "if anything, the contrary." This source is simulation-only and does not measure effect inflation directly.

For evidence synthesis, the sources are descriptive. Meta-analyses whose accumulation of trials depends on strategic decisions have no valid type I error control. Reanalyzing an individual trial in an unplanned meta-analysis also breaks control, because the individual analysis has already spent the alpha. Neither claim comes with a measured magnitude or frequency, so the size of this problem in practice is not established here.

Where the evidence splits

No contradictions in the current sources.

Thresholds and dose-response

Selection threshold on the marker: in the insomnia trial, the observed drug effect was 39% larger than the unselected effect at a threshold of 15 minutes. It was 106% larger at 20 minutes. The source states that a larger threshold gives a more prominent effect.
Selection at a marker value of at least 0.6, with a true effect of zero: the probability of claiming a drug effect was about 98%, against a nominal 5%.
Allocation design under selection bias: type I error was 38% to 39% for PBD(2) and 8% to 9% for BSD(3). The source describes a gradient in which more random designs are less susceptible.
Sample size: the inflation does not change with sample size. The chance that it is detected rises with sample size.

Open empirical questions

How large is selection bias in real randomized trials, as opposed to simulated bias effects? The sources here give no measured estimate.
How often does treatment change the marker–outcome correlation in real selected-population trials? One insomnia trial is the only real-data case on this page.
How much do accumulation bias and unplanned reanalysis inflate the false positive rate in published meta-analyses? The sources state the problem without a magnitude.
Does the order-dependence seen in simulated Phase I designs inflate or distort effect estimates in actual dose-finding trials?

Findings

Selection bias increases the type I error rate under every randomization procedure tested

confidence — direction: highconfidence — magnitude: highconsensus: thin. This is one simulation study, and the weight leans toward inflation under all procedures.

mechanism: a selection bias effect in who is assigned to each arm makes the arms differ by more than treatment, so tests reject a true null more often than their nominal level.
effect: increases type I error. CRD reached about 7% against a nominal 5%. The inflation depends on the size of the bias effect and not on sample size.
conditions / moderators: depends on the magnitude of the bias effect. The additional simulations behind the sample-size result are not shown in the paper.

Vance W. Berger (2021), BMC Medical Research Methodology · doi:10.1186/s12874-021-01303-z · https://bmcmedresmethodol.biomedcentral.com/track/pdf/10.1186/s12874-021-01303-z (retrieved 2026-10-06)

Less random allocation designs show larger type I error inflation under selection bias

confidence — direction: highconfidence — magnitude: mediumconsensus: thin. One simulation study agrees with prior theory, and the weight leans toward more random designs being less susceptible.

mechanism: deterministic-heavy designs, such as permuted blocks with a small block size, make upcoming assignments easier to predict. This gives more room for selection bias.
effect: increases type I error. PBD(2) reached 38% to 39%. BSD(3) reached 8% to 9%. MTI and biased coin designs were about 6% to 9%. A cited theoretical literature reports serious inflation for small blocks, with no magnitude given.
conditions / moderators: the ranking was consistent across three tests. Effect sizes depend on the bias effect parameter.

Vance W. Berger (2021), BMC Medical Research Methodology · doi:10.1186/s12874-021-01303-z · https://bmcmedresmethodol.biomedcentral.com/track/pdf/10.1186/s12874-021-01303-z (retrieved 2026-10-06)

Patient selection on a baseline marker increases the observed treatment effect when treatment changes the marker–outcome correlation

confidence — direction: highconfidence — magnitude: mediumconsensus: thin. This is one author's analytic and simulation work plus one real trial, and the weight leans toward inflation.

mechanism: treatment changes the correlation between the selection marker and the outcome differently than control does. Selecting on the marker then produces a gap between arms that is not a treatment effect.
effect: increases the observed effect. In the insomnia trial it was 39% larger at a 15-minute threshold and 106% larger at 20 minutes. With a true effect of zero in a simulation, the probability of a claimed drug effect rose from 5% to about 98%.
conditions / moderators: a stricter threshold gives a larger effect. The inflation does not depend on sample size, but its chance of being detected rises with sample size. It depends on covariance differences between arms that are often unobservable before the trial ends.

Junshui Ma (2020), arXiv · doi:10.48550/arxiv.2001.02036v1 · https://arxiv.org/pdf/2001.02036v1 (retrieved 2026-10-07)

Early cohorts increase their influence on dose selection in long-memory Phase I designs, and early settling shows no link to better dose choice

confidence — direction: highconfidence — magnitude: mediumconsensus: thin. This is one simulation study.

mechanism: a "winner-takes-all" dose assignment rule gives early cohorts a disproportionately large influence. Run-to-run variability then follows the order in which participants enter.
effect: variability in the number of cohorts allocated to the true dose follows enrollment order. Early settling on a dose shows little evidence of better maximum-tolerated-dose performance, and if anything the reverse. No magnitude is given in the claims.
conditions / moderators: simulation-only. The early-settling result comes from visual inspection of multiple scenarios.

Assaf P. Oron (2012), arXiv · doi:10.1177/1740774512469311 · https://arxiv.org/pdf/1202.4962v2 (retrieved 2026-10-06)

Data-driven accumulation of trials and unplanned reanalysis remove valid type I error control in meta-analysis

confidence — direction: highconfidence — magnitude: lowconsensus: thin. A single author states this as a property of the procedure.

mechanism: when trials are accumulated on strategic decisions, accumulation bias enters. When an individual trial analysis is reanalyzed in an unplanned meta-analysis, the alpha is already spent on the trial analysis.
effect: removes valid type I error control. No magnitude is quantified.
conditions / moderators: not established. The sources describe design properties and do not measure how often or how badly this occurs.

Judith ter Schure (2021), arXiv · doi:10.12688/f1000research.74223.2 · https://arxiv.org/pdf/2109.12141v2 (retrieved 2026-10-07)