Empirical Change in Published Estimates After Common-Support Restriction

confidence: high · 3 sources

⚠
Working research — not a recommendation.

This page records what the evidence says a policy does, not what anyone should do about it. It has been synthesized from the cited sources but has not been independently reviewed.

What the evidence says

The sources on this page measure how an estimate changes when the analysis is restricted to a narrower set of comparable units. They cover three settings: indirect treatment comparisons, genetic analyses of sequential educational transitions, and meta-analyses limited to those with enough information. The designs differ, so the findings are not pooled.

In one indirect comparison of paracetamol plus codeine, the adjusted indirect estimate based on all trials was −1.16 (95% CI −6.95 to 4.64). The direct estimate was 7.28 (3.69 to 10.87), and the two differed at P=0.02. Restricting to trials that used 600-650 mg paracetamol and 60 mg codeine moved the adjusted indirect estimate to 5.72 (−5.37 to 16.81). That is no longer different from the direct estimate. The source attributes the original gap to many placebo-controlled trials using lower doses (300 mg paracetamol, 30 mg codeine). The authors conclude that trial similarity should be assessed carefully. Only this one case is examined in detail, so it is unknown whether other trial characteristics behave the same way.

In the genetic analyses, estimates fall at each later transition, where the at-risk group is progressively narrower. Observed-scale common-variant heritability (h2SNP) is about 0.14 for high school to bachelor's or higher (HS→BSc+). It falls to about 0.10 at BSc→MSc+ and about 0.07 at MSc→PhD. Polygenic index discrimination (Tjur's R2) is about 0.05 at HS→BSc+, about 0.03 at BSc→MSc+, and below 0.01 at MSc→PhD. Genetic correlation with the external EA4 educational attainment GWAS falls from 0.92 to 0.82 to 0.38 over the same transitions. Correlation with intelligence falls from 0.61 to 0.56 to 0.28. The MSc→PhD estimates are less precise, with an effective sample size of 8,185.

The source offers a selection mechanism for this attenuation. Selection into an already-selected at-risk group may narrow the residual variation in education-related characteristics available to distinguish further progression. The authors say the design cannot separate how much of the decline is selection and how much is true change in effects across milestones. The gap between twin-based and SNP-based heritability narrows from about 0.38 at high-school completion to about 0.11 at PhD on the liability scale. The twin and SNP estimates come from different cohorts, so the narrowing may reflect cohort differences and not only method.

A third body of evidence concerns meta-analyses that have not reached their required information size. It does not involve common-support restriction as such, but it shows the same pattern of published estimates changing when the underlying sample changes. Early meta-analyses often show large effects that shrink in later updates. A large simulation of random-effects meta-analyses shows a considerable risk of overestimating the effect before the required information size is reached. Among neonatal intervention meta-analyses, about 25% to 30% were false positive results. Naive 95% confidence intervals applied before that point are too narrow.

Where the evidence splits

No contradictions in the current sources.

Thresholds and dose-response

No threshold established in the current sources.

Open empirical questions

Findings

Restricting an indirect comparison to trials with similar dosing decreases the gap between direct and indirect estimates.

confidence — direction: highconfidence — magnitude: mediumconsensus: thin — one worked example, with the authors stating that trial similarity must be assessed.
mechanism
The unrestricted indirect comparison pooled placebo-controlled trials that used lower doses (300 mg paracetamol, 30 mg codeine). Restricting to 600-650 mg paracetamol and 60 mg codeine removes that dose heterogeneity.
effect
The adjusted indirect estimate moved from −1.16 (95% CI −6.95 to 4.64) to 5.72 (95% CI −5.37 to 16.81). The direct estimate was 7.28 (3.69 to 10.87). The discrepancy fell from P=0.02 to non-significance.
conditions / moderators
Shown for one paracetamol-codeine example. It is not established whether restriction on other intervention characteristics behaves the same way.

Restricting to later educational transitions decreases observed-scale common-variant heritability.

confidence — direction: highconfidence — magnitude: highconsensus: thin — one study.
mechanism
Selection into progressively narrower at-risk groups may restrict residual variation in education-related characteristics, which can attenuate the genetic signal without true effect heterogeneity.
effect
h2SNP is about 0.14 at HS→BSc+, about 0.10 at BSc→MSc+, and about 0.07 at MSc→PhD.
conditions / moderators
The source describes an inverse-U pattern across transitions, peaking at HS→BSc+. The design cannot separate selection from true milestone-specific change.

Restricting to later educational transitions decreases polygenic index discrimination.

confidence — direction: highconfidence — magnitude: highconsensus: thin — one study.
mechanism
Not separately established. The source notes that the decline may reflect lost predictive signal or selection-induced reduction in residual variance and case-proportion imbalance.
effect
Tjur's R2 is about 0.05 at HS→BSc+, about 0.03 at BSc→MSc+, and below 0.01 at MSc→PhD. At Comp→HS+ it is about 0.03.
conditions / moderators
Models were adjusted for sex, birth year, and ancestry principal components. Inverse-U pattern across transitions.

Restricting to later educational transitions decreases genetic correlations with external educational attainment and intelligence GWAS.

confidence — direction: highconfidence — magnitude: mediumconsensus: thin — one study.
mechanism
The source suggests selection into an already-selected at-risk group may restrict residual variation. This is not tested directly.
effect
The correlation with EA4 is 0.92 at HS→BSc+, 0.82 at BSc→MSc+, and 0.38 at MSc→PhD. The correlation with intelligence is 0.61, 0.56, and 0.28 over the same transitions.
conditions / moderators
MSc→PhD estimates are markedly less precise (effective sample size 8,185).

Restricting by milestone in cumulative analyses decreases the gap between twin-based and SNP-based heritability.

confidence — direction: highconfidence — magnitude: mediumconsensus: thin — one study.
mechanism
Not established.
effect
On the liability scale, the gap narrows from about 0.38 at high-school completion to about 0.11 at PhD.
conditions / moderators
Twin and SNP estimates come from different cohorts with different ages, sampling, milestone prevalence, and populations. The narrowing may reflect cohort variation and not method alone.

Meta-analyses that have not reached the required information size show larger effects than later updates, and 25% to 30% of neonatal meta-analyses were false positive.

confidence — direction: highconfidence — magnitude: mediumconsensus: thin — one methodological source, though it cites other studies.
mechanism
Before the required information size is reached, traditional unadjusted 95% confidence intervals are too narrow. Chance findings then look precise and can exclude the null.
effect
Early meta-analyses often show unrealistically large effects that become considerably lower in updates. A large simulation shows a considerable risk of overestimating the effect before the required information size is reached. Among neonatal intervention meta-analyses, about 25% to 30% were false positive.
conditions / moderators
The sources give no specific effect magnitude for the simulation. The 25% to 30% figure applies to neonatal interventions, and the sample reviewed is not described in the claim.