Rogue-dimension analysis

methods / Theoretical / Analytical / Geometric analysis / Rogue-dimension analysis

Techniqueadvanced

Decomposes cosine similarity into a per-dimension contribution, identifies the handful of high-variance, far-from-origin dimensions that dominate it, tests those dimensions' actual behavioral importance via KL-divergence ablation, and corrects for the mismatch via per-dimension z-scoring (standardization) before trusting any cosine-similarity-based analysis.

Used in (3 observations)

Outlier dimensions persist and can solve GLUE tasksmeasured

structure: Linear Separability, Anisotropy · models: BERT-base-uncased, ALBERT-base-v1, DistilBERT-base-uncased, RoBERTa-base, GPT-2 Small, Pythia-70M, Pythia-160M, Pythia-410M · paper: Outlier Dimensions Encode Task-Specific Knowledge

A few rogue dimensions drive transformer anisotropymeasured

structure: Anisotropy · models: BERT-base-cased, RoBERTa-base, GPT-2-small, XLNet-base-cased, word2vec (Google News, 300d) · paper: All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality

Dominant-dimension variance decides cosine vs rank metricsmeasured

structure: Anisotropy · models: all-MiniLM-L6-v2, all-MiniLM-L12-v2, all-mpnet-base-v2, paraphrase-mpnet-base-v2, BGE-base-en-v1.5, BGE-large-en-v1.5, E5-large-v2, multilingual-e5-large, e5-mistral-7b-instruct, SFR-Embedding-Mistral, BERT-base-uncased, RoBERTa-base, ELECTRA-base, mBERT (BERT-base, Multilingual Cased), GPT-2-small, Pythia-410M, Qwen2.5-1.5B, Qwen2.5-7B, Mistral-7B · paper: Anisotropy Decides Cosine vs. Rank Metrics for Text Embeddings