MATH · IN · MODELS

XLM-RoBERTa

Meta AI (FAIR)

Structures found in this family (4)

Linear Subspace, Anisotropy, Linear Direction, Attention reference frame (sink-token anchor configuration)

By model (3)

XLM-RoBERTa base · 270M
Linear Subspace, Anisotropy, Linear Direction
Multilingual S-BERT (sentence-transformers/xlm-r-100langs-bert-base-nli-stsb-mean-tokens)
Anisotropy
XLM-RoBERTa large · 550M
Attention reference frame (sink-token anchor configuration)

Observations (7)

After mean-centering, 88 languages share one XLM-R subspace
Mean-pooled sentence embeddings of XLM-R and mBERT are highly anisotropic, with a single outlier dimension dominating the mean cosine; multilingual S-BERT is much more isotropic
A language difference-of-means direction causally steers target-language output
A naturalistic causal ATE direction nearly coincides with the top PCA direction
Probing accuracy collapses under a naturalistic counterfactual test
Attention sinks form three position-encoding-linked reference-frame configurations
A centroid-estimated linear grammatical-gender direction in frozen Spanish/French encoders gives a better leakage-reduction vs. occupation-preservation trade-off than SVM or LDA estimators, and controlled templates give cleaner noun-level reduction than natural contexts

Papers

The Geometry of Multilingual Language Model Representations (2022), Exploring Anisotropy and Outliers in Multilingual Language Models for Cross-Lingual Semantic Sentence Similarity (2023), Naturalistic Causal Probing for Morpho-Syntax (2023), What are you sinking? A geometric approach on attention sink (2025), Estimating Grammatical Gender Directions in Contextual Embeddings under Controlled and Natural Contexts (2026)