Balance Diagnostics Beyond Standardized Means

confidence: high · 2 sources

What the evidence says

The only source here that speaks directly to balance diagnostics is a 2026 arXiv paper by Usef Faghihi. It reports two benchmarks: a fully synthetic exact benchmark and a semi-synthetic benchmark built on real covariates. In both, weak ignorability fails. Balancing observed covariates nearly eliminates standardized mean differences, yet the coordinate-mean average treatment effect remains biased. A near-zero standardized mean difference therefore does not show that the treatment effect estimate is unbiased. The claim ties the bias to altered distributional structure. Only the abstract was accessible, so the mechanism, the size of the bias, and the conditions under which it appears are not available on this page.

Two other claims come from a different field. Lewandowsky (2008) analyzes the word-length effect in memory research. They describe it as inherently correlational. The experimenter selects items that differ in presumed articulation time, measures actual articulation times, and uses those to predict memory performance. Lewandowsky also argues that long and short words differ on many attributes besides articulation duration, which creates confounds. The claims cite Davis (2005) as identifying nearly 50 potential word features. These claims show the same general problem, that a comparison can look clean on the intended variable while other attributes differ. They are not about covariate balance in treatment-effect estimation, and they give no diagnostic.

No claim here describes an alternative to standardized mean differences. The sources do not cover distributional comparisons, higher-moment checks, or any test that detects the bias in the Faghihi benchmarks. The page documents a failure of one diagnostic. It does not document a replacement.

Where the evidence splits

No contradictions in the current sources.

Thresholds and dose-response

No threshold established in the current sources.

Open empirical questions

How large is the remaining bias in the coordinate-mean average treatment effect once standardized mean differences are near zero, and how does it change with the degree of distributional mismatch?
Which diagnostics beyond standardized means detect the bias in the Faghihi benchmarks, and how often do they flag it?
Does the pattern in the two synthetic and semi-synthetic benchmarks appear in real observational datasets with known ground truth?
Do the uncontrolled attributes identified in the word-length literature change measured outcomes, and by how much?

Findings

Balancing observed covariates decreases standardized mean differences to near zero while the coordinate-mean average treatment effect stays biased

confidence — direction: highconfidence — magnitude: mediumconsensus: thin — one source, abstract only

mechanism: the claim states that distributional structure is altered and weak ignorability fails; the mechanism beyond that is not available, because only the abstract was accessible
effect: standardized mean differences approach zero; treatment effect bias persists; the size of the bias is not stated
conditions / moderators: shown in a fully synthetic exact benchmark and a real-covariate semi-synthetic benchmark, both with weak ignorability failing; behavior in observational data outside these benchmarks is not established

Usef Faghihi (2026), arXiv · doi:10.48550/arxiv.2606.01184v2 · https://arxiv.org/pdf/2606.01184v2 (retrieved 2026-10-01)

The word-length effect shows no measured causal effect on memory decay because its design is correlational

confidence — direction: highconfidence — magnitude: highconsensus: thin — one source, a conceptual argument with descriptive analysis

mechanism: the experimenter selects items that differ in presumed articulation time and uses measured articulation times to predict memory performance, so the design cannot isolate a causal path
effect: no measured causal effect; correlation only
conditions / moderators: the source's Bayesian diagnosticity argument, that P(WLE|decay) ≠ P(WLE|no decay), would need to be accepted; this concerns memory research, not covariate balance

Stephan Lewandowsky (2008), Psychonomic Bulletin & Review · doi:10.3758/pbr.15.5.875 · https://link.springer.com/content/pdf/10.3758/PBR.15.5.875.pdf (retrieved 2026-10-02)

Long and short words differ on attributes beyond articulation duration, which introduces uncontrolled confounds

confidence — direction: highconfidence — magnitude: mediumconsensus: thin — one source, descriptive analysis

mechanism: items chosen for length also differ in other features, which confounds the variable of interest, the time between rehearsals of a given item
effect: confounds present; magnitude unknown
conditions / moderators: whether the uncontrolled variables beyond those identified change recall is not established; Davis (2005) identified nearly 50 potential word features

Stephan Lewandowsky (2008), Psychonomic Bulletin & Review · doi:10.3758/pbr.15.5.875 · https://link.springer.com/content/pdf/10.3758/PBR.15.5.875.pdf (retrieved 2026-10-02)