Whitening transformation

methods / Theoretical / Analytical / Geometric analysis / Whitening transformation

Techniqueintermediate

Classical closed-form linear transformation — subtract the mean, then decorrelate and rescale every axis to unit variance via the eigendecomposition of the covariance matrix — applied as a post-hoc fix for anisotropic sentence embeddings. Unlike the learned normalizing-flow fix, this requires no training: a direct, exact linear map computed once from the embedding covariance.

Used in (4 observations)

Dimension-level activation outliers, measured as the ratio gamma of mean norm to per-token variation, predict sparse-autoencoder feature death across 454 model-layer combinations spanning language, vision, protein and genomic modelsmeasured

structure: Anisotropy · models: GPT-2, Pythia-410M, Pythia-70M, ESM-2 (3B), ESM-2 (650M), ESM-2 (35M), ProGen2-base, DINOv3 ViT-B/16, DINOv2 ViT-B · paper: On the Relationship Between Activation Outliers and Feature Death in Sparse Autoencoders

Mean-pooled sentence embeddings of XLM-R and mBERT are highly anisotropic, with a single outlier dimension dominating the mean cosine; multilingual S-BERT is much more isotropicmeasured

structure: Anisotropy · models: XLM-RoBERTa base, mBERT (BERT-base, Multilingual Cased), Multilingual S-BERT (sentence-transformers/xlm-r-100langs-bert-base-nli-stsb-mean-tokens) · paper: Exploring Anisotropy and Outliers in Multilingual Language Models for Cross-Lingual Semantic Sentence Similarity

Whitening removes anisotropy and improves sentence similaritymeasured

structure: Anisotropy · models: BERT-base-uncased, RoBERTa-base, DistilBERT-base-uncased, LaBSE · paper: WhiteningBERT: An Easy Unsupervised Sentence Embedding Approach

Whitening matches BERT-flow at far lower dimensionmeasured

structure: Anisotropy · models: BERT-base-uncased, BERT-large-uncased · paper: Whitening Sentence Representations for Better Semantics and Faster Retrieval