XLM-RoBERTa

Meta AI (FAIR)

Structures found in this family (3)

Linear Subspace, Linear Direction, Attention reference frame (sink-token anchor configuration)

By model (2)

XLM-RoBERTa base · 270M

Linear Subspace, Linear Direction

XLM-RoBERTa large · 550M

Attention reference frame (sink-token anchor configuration)

Observations (5)

After mean-centering, 88 languages share one XLM-R subspace

A language difference-of-means direction causally steers target-language output

A naturalistic causal ATE direction nearly coincides with the top PCA direction

Probing accuracy collapses under a naturalistic counterfactual test

Attention sinks form three position-encoding-linked reference-frame configurations

Papers

The Geometry of Multilingual Language Model Representations (2022), Naturalistic Causal Probing for Morpho-Syntax (2023), What are you sinking? A geometric approach on attention sink (2025)