Distributional Balance Diagnostics for Causal Inference

confidence: high · 3 sources

⚠
Working research — not a recommendation.

This page records what the evidence says a policy does, not what anyone should do about it. It has been synthesized from the cited sources but has not been independently reviewed.

What the evidence says

The sources on this page agree on one mechanism. When treated and control units differ in their covariate distributions, estimates depend on how the model extrapolates. Ho (2007) names the source: control units that sit far outside the range of the treated units. The model then extrapolates over a range that holds no treated and control units together. It becomes sensitive to the control units that do not look like the treated ones. This is a descriptive argument, shown in an illustration. The source does not test it across multivariate data.

Ho (2007) also shows the reverse in a simulation. After deleting the poorly matched units, the data set is highly balanced. A linear and a quadratic model then give essentially identical effects, both about 0.001, against a true effect of 0. Balancing therefore decreases the dependence of estimates on functional form in that example. The evidence is one simulated illustration. Whether it holds with many covariates and more complex imbalance is not established here.

Hainmueller (2018) finds the same problem for conditional marginal effects in interaction models. Where the moderator has no or only sparse data in one treatment group, the estimates rest on extrapolation or interpolation of the functional form. The estimates are then fragile and model dependent. In the replications of 46 interaction effects from 22 published studies, this extrapolation is common. Articles typically report effects across the whole range of the moderator, which often includes large intervals with no or very few observations.

The sources also describe how to diagnose balance. Hainmueller (2018) says to compare the distribution of the moderator across treatment groups. The analyst then checks the range of values with enough data points. Box plots can show this: median, 25th and 75th percentiles, and 5th and 95th percentiles by treatment level. Binning the moderator by its observed support keeps effects at typical values and avoids excessive extrapolation. These are methods shown on simulated and published data. The sources report no measure of how often they change published conclusions.

Aronow (2015) addresses a different use of balance tests: dropping subjects based on a post-treatment manipulation check. A balance test can verify the plausibility of a sufficient condition for unbiased estimation after dropping. The converse does not hold. Balance on observed covariates is not sufficient to justify discarding subjects. This is a theoretical result. It sets a limit on what a balance test shows.

Where the evidence splits

No contradictions in the current sources.

Thresholds and dose-response

No threshold established in the current sources.

Open empirical questions

Findings

Balancing treatment and control covariate distributions decreases the sensitivity of causal estimates to model specification

confidence — direction: highconfidence — magnitude: mediumconsensus: thin — one illustration, no contrary evidence
mechanism
Deleting units outside the region of overlap removes the extrapolation that makes estimates depend on functional form.
effect
Decreases model dependence. In the simulated example, linear and quadratic models both give about 0.001 after matching, against a true effect of 0.
conditions / moderators
Shown in a one-covariate simulation. Generalization to many covariates and complex imbalance is not established.

Control units far outside the range of treated units increase model dependence

confidence — direction: highconfidence — magnitude: mediumconsensus: thin — descriptive argument, no contrary evidence
mechanism
The model extrapolates over a range with no treated and control units together, so it is sensitive to dissimilar control units.
effect
Increases model dependence. No magnitude is given.
conditions / moderators
Illustrated in one dimension. Whether it holds in multivariate cases is not established.

Lack of common support makes conditional marginal-effect estimates fragile and model dependent, and such extrapolation is common in published work

confidence — direction: highconfidence — magnitude: mediumconsensus: thin — one author's replication set, no contrary evidence
mechanism
Where the moderator has no or sparse data in a group, estimates rely on extrapolation or interpolation of the functional form.
effect
Estimates are fragile and model dependent. Extrapolation is common across 46 replicated interaction effects from 22 published studies. No magnitude is given.
conditions / moderators
Applies to conditional marginal effects in interaction models. Articles typically report the whole moderator range, which often includes intervals with no or very few observations.

Comparing moderator distributions by treatment group, with box plots and binning, identifies common support and limits extrapolation

confidence — direction: highconfidence — magnitude: mediumconsensus: thin — methodological exposition from one source
mechanism
The comparison shows the range of moderator values with enough observations. Bins built on observed support estimate effects at typical values.
effect
Diagnostic and estimation methods. Binning prevents excessive extrapolation. No effect size is given.
conditions / moderators
Demonstrated on simulated and published data. Performance in other settings is not established.

A balance test checks a sufficient condition for dropping subjects, but balance on observed covariates does not justify the drop

confidence — direction: highconfidence — magnitude: highconsensus: thin — one theoretical source
mechanism
If treatment is independent of the manipulation check results, the probability that the check outcomes are equal across treatment levels is 1. A balance test can verify this. The converse does not hold.
effect
A balance test provides information about the identification condition. It does not by itself justify discarding subjects.
conditions / moderators
Applies to dropping subjects on post-treatment manipulation checks. Theoretical result, not an empirical estimate.