A naturalistic causal ATE direction nearly coincides with the top PCA direction

Amini et al. estimate the average treatment effect of grammatical gender and number on mBERT, XLM-RoBERTa-base, and GPT2-small-Spanish using dependency-tree-reinflected naturalistic counterfactuals [amini-etal-2023-naturalistic] The first principal component of the paired representations has cosine similarity ~0.99 with the estimated ATE direction across all three models [amini-etal-2023-naturalistic] In BERT/RoBERTa this PC explains ~20% of paired variance, while in GPT2-small-Spanish over 50% concentrates in the first 1-2 components, a starker one-dimensional encoding [amini-etal-2023-naturalistic] The paired causal estimator is far more stable across treebanks than a naive correlational one and better reconstructs the true counterfactual distribution [amini-etal-2023-naturalistic]