Positivity Violations and Covariate-Balance Diagnostics in Applied Studies

What the evidence says

Causal estimates from observational data rest on two conditions the sources treat as baseline. One is positivity: every unit has a propensity score strictly between 0 and 1, so everyone has a chance of receiving the treatment. The other is no unmeasured confounding. The two together make up strong ignorability. Propensity score methods also require that overlap and covariate balance hold. Where positivity fails, causal effects can only be estimated for the subpopulations that remain, so inference may have to focus on regions of covariate overlap. Positivity violations can sometimes be diagnosed empirically, but near-violations may be hard to detect.

Covariate imbalance is the measured link between design and bias. A theoretical decomposition shows that bias in causal comparisons corresponds directly to distributional imbalance of covariates between treatment groups. A weighted energy distance can therefore estimate how much bias a given set of weights and dataset will produce. It can also compare weights from different methods, such as propensity score modeling or empirical calibration. Separately, a simulation demonstration and cited literature find that even mild propensity score misspecification can produce large bias. In indirect treatment comparisons, unadjusted anchored comparisons were biased when covariates were imbalanced. Population-adjusted methods (MAIC and STC) reduced that bias, at the cost of higher variance. They did so most clearly where effect modifiers had strong effects. The magnitudes are not given in the claims.

Applied balance testing looks weak by comparison. A survey of 150 articles from top political science journals (APSR, AJPS, JOP, 2000–2019) found no examples of efforts to measure the prognosis of the covariates used in balance tests. Only 18% of balance tests used the pre-treatment value of the dependent variable as a covariate. The same paper argues that balance is uninformative about key identification conditions when measured covariates are unrelated to potential outcomes. In simulations, prognosis-weighted tests gained power and specificity over standard unweighted tests, and unweighted multivariate tests sacrificed power, specificity, or both. The simulations show a consistent direction, but the size of the gain varies with the data-generating process. The survey covers political science only, so the claims here do not say how common the omission is in other fields.

Weighting estimators are fragile when overlap is limited. In 36 simulation scenarios with 5,000 replications each, inverse probability weighting (IPTW) had the worst root mean squared error in the most challenging settings. That error rose as scenario complexity grew. A second source says IPTW can produce extreme weights when overlap is limited or misspecification is plausible. These often require trimming or alternative targets. Doubly robust estimation with an outcome regression adjusted for all confounders decreased sensitivity to the weighting scheme, despite functional-form misspecification. The authors of one covariate-balancing method (kernel optimal matching) are described as not recommending the doubly robust estimator, citing practical positivity violations and bias risk from misspecified outcome and treatment models. The cited papers do not evidence that statement within the claims here.

One applied cohort shows what diagnostics look like when overlap is adequate. Stabilized weights had a mean of 1.02 (SD 0.18) and a maximum truncated weight of 2.94. Triglyceride z-score estimates moved only from +0.17 to +0.19 across truncation thresholds of 0.5%/99.5%, 1%/99%, and 2%/98%. The maximum post-weighting standardized mean difference stayed at or below 0.08. The authors report adequate measured overlap without extreme positivity violations. This is one cohort, so it cannot show how often such results occur.

Where the evidence splits

Two sources differ on how far balance metrics can stand in for evaluating an estimator. One derives a link between distributional imbalance and bias, and proposes a weighted energy distance to gauge the expected bias of a set of weights. The other says it would be ill-advised to appraise balancing methods by a single balance metric, because some methods target balance only indirectly, for example by minimizing worst-case bias rather than matching moments. It assesses them by treatment effect estimation accuracy instead. The claims state the second source's reason. They do not say whether the energy distance avoids that objection, so the split is unexplained by the sources on this page.

Thresholds and dose-response

No threshold established in the current sources.

Open empirical questions

• How often do near-violations of positivity go undetected in applied studies, and how much bias do they leave?
• Does prognosis-weighted balance testing gain power outside political science, and by how much, when applied to real datasets rather than simulations?
• How does the weighted energy distance perform against outcome-based assessment for methods that target balance indirectly?
• What share of applied studies outside the sampled journals measure covariate prognosis or include the pre-treatment outcome in balance tests?