Propensity Score Matching Covariate Balance Results (Abdulsalam Thesis Table 5.3)

confidence: high · 4 sources

⚠
Working research — not a recommendation.

This page records what the evidence says a policy does, not what anyone should do about it. It has been synthesized from the cited sources but has not been independently reviewed.

What the evidence says

None of the sources on this page is the Abdulsalam thesis or its Table 5.3. The evidence below comes from one applied study and three methods papers on propensity score matching. It describes how covariate balance behaves after matching in general, not what that thesis table reports.

Across the applied and illustrative examples, matching decreases measured covariate imbalance. In You (2023), the standardized deviation of all covariates after matching was less than 10%. In Austin's analysis of 9104 patients with acute myocardial infarction, the largest absolute standardized difference was 0.030 in the matched sample and 0.910 in the unmatched sample. In Thoemmes's illustrative example, the largest remaining standardized difference after matching was d = .09, at the interaction between socio-economic status and problems with parents. Thoemmes's global L1 imbalance statistic fell from .938 in the unmatched sample to .919 in the matched sample. That is a small drop, and the source reads it as improved overall balance.

Formal balance tests agree with these measures. In You (2023), the p-values for covariate balance ranged from 0.453 to 0.966, so the test did not reject the hypothesis of no systematic difference. Nearest neighbour, radius and kernel matching all passed. In Thoemmes, the overall multivariate test gave χ2(8) = 13.89, p = .09, which is above the .05 threshold.

Passing these checks does not show that the propensity score model is correct. Austin's Monte Carlo simulations found that balanced covariate means do not imply a correctly specified model. Bias was greater when matching on a mis-specified score than on a correct one, even though the means were balanced. Austin reports that other diagnostics do separate the two cases. When the model was mis-specified, the variance ratio of a baseline covariate was moderately different from one. It was close to one when the model was correct. Standardized differences of interaction terms between covariates were also larger under mis-specification. Comparing the distribution of estimated propensity scores between groups was uninformative about balance and about specification.

Matching also changes the outcome estimate when confounding is present. In Thoemmes's example, the effect of moving out on neuroticism fell from d = -.41 in the unmatched sample to d = -.11 in the matched sample (t(2674) = -1.93, p = .053). This example is illustrative, not a field result.

There is no agreed cutoff for "balanced". Li (2011) states that there is no widely accepted guidance on how small a standardized difference must be. Li also argues that with matching weights, statistical tests can check balance, provided the propensity score model is correct.

Where the evidence splits

No contradictions in the current sources.

Thresholds and dose-response

No threshold established in the current sources.

Open empirical questions

Findings

Propensity score matching decreases measured covariate imbalance

confidence — direction: highconfidence — magnitude: mediumconsensus: settled. All four examples point the same way.
mechanism
matching on the propensity score produces a sample in which the means and prevalences of baseline covariates are similar between treated and untreated subjects.
effect
decreases imbalance. Standardized deviation of all covariates was less than 10% after matching (You). The largest absolute standardized difference was 0.030 matched versus 0.910 unmatched (Austin, 9104 patients). The largest remaining standardized difference was d = .09, down from a pre-matching imbalance in Thoemmes's example. L1 fell from .938 to .919 (Thoemmes).
conditions / moderators
Austin's result is for an adequately specified propensity score model. Thoemmes's example is simulation-based. The L1 drop is small.

Formal balance tests do not reject balance after matching

confidence — direction: highconfidence — magnitude: highconsensus: settled for these examples. The two sources report different test types.
mechanism
tests compare covariates between matched treated and control groups. A large p-value means no detectable systematic difference.
effect
no measured imbalance on the tests reported. Covariate balance p-values ranged from 0.453 to 0.966 for nearest neighbour matching (You). Radius and kernel matching also passed. The multivariate test gave χ2(8) = 13.89, p = .09 (Thoemmes).
conditions / moderators
Li states that tests of balance presume the propensity score model is correct.

Balanced covariate means do not show that the propensity score model is correctly specified

confidence — direction: highconfidence — magnitude: mediumconsensus: thin. One author and one simulation design support it.
mechanism
a mis-specified model can balance means while leaving other features of the covariate distributions unbalanced. Variance ratios and interaction terms show the remaining imbalance.
effect
bias in the treatment effect estimate was greater under a mis-specified model than a correct one, despite balanced means. The variance ratio was moderately different from one under mis-specification and close to one under correct specification. Interaction-term standardized differences were larger under mis-specification.
conditions / moderators
results are from Monte Carlo simulations. The interaction result is reported for Scenarios B and C. Comparing propensity score distributions between groups does not distinguish correct from mis-specified models.

No accepted cutoff defines adequate standardized-difference balance

confidence — direction: highconfidence — magnitude: lowconsensus: thin. One source states the gap.
mechanism
not applicable. This is a gap in measurement guidance.
effect
no measured effect to report. The standardized difference is the general recommendation, but there is no widely accepted guidance on how small is small enough.
conditions / moderators
not established.

Matching decreases the estimated treatment effect when confounding is present

confidence — direction: highconfidence — magnitude: lowconsensus: thin. One illustrative example.
mechanism
matching removes the part of the unadjusted difference that comes from imbalanced covariates.
effect
decreases the estimate. The standardized mean difference went from d = -.41 unmatched to d = -.11 matched, and the matched test gave t(2674) = -1.93, p = .053.
conditions / moderators
a single simulation-based illustrative example. The source says the drop is not purely a result of the smaller matched sample.