Mean-pooled sentence embeddings of XLM-R and mBERT are highly anisotropic, with a single outlier dimension dominating the mean cosine; multilingual S-BERT is much more isotropic

Mean cosine of random sentence pairs on Tatoeba (layer 8) is 0.92 for XLM-R, 0.73 for mBERT (cased) and 0.35 for the multilingual S-BERT; XLM-R dimension 588 alone contributes 0.77 of the 0.92 [hammerl-etal-2023-anisotropy-and-outliers-in-multilingual-models-for-cross-lingual-sentence-similarity] On Wikipedia sentences in the output layer the values are 0.99 (XLM-R, dimension 588 contributes 0.89), 0.61 (mBERT) and 0.27 (S-BERT); outlier dimensions differ from layer to layer [hammerl-etal-2023-anisotropy-and-outliers-in-multilingual-models-for-cross-lingual-sentence-similarity] For mBERT and S-BERT anisotropy is lowest for English and Spanish and highest for Swahili, roughly increasing as pre-training data shrinks (mBERT: en 0.49, sw 0.69) [hammerl-etal-2023-anisotropy-and-outliers-in-multilingual-models-for-cross-lingual-sentence-similarity] Zeroing dimension 588 lifts XLM-R Tatoeba accuracy only from 50.35 to 52.99, whereas whitening or cluster-based isotropy enhancement brings anisotropy to about 0 and accuracy to 70.03 and 69.01 [hammerl-etal-2023-anisotropy-and-outliers-in-multilingual-models-for-cross-lingual-sentence-similarity]