Definition
A set of vectors is isotropic if it is directionally uniform — informally, no direction is preferred over any other. Ethayarajh (2019) operationalizes the degree of anisotropy of a representation space (e.g. one layer of a contextualizing model) via the expected cosine similarity between two uniformly-randomly-sampled representations from that space:
where is the set of all word occurrences and maps an occurrence to its representation in layer . This statistic lies in . A value near indicates one specific, one-sided form of extreme anisotropy — almost every pair of vectors points in nearly the same direction (a single narrow cone).
The mean cosine is one particular, one-sided probe — not a full isotropy test. Two cautions:
- does not imply isotropy. Counterexample: put half the unit mass along and half along . The cloud is maximally anisotropic (all mass on one line), yet because same-sign and opposite-sign pairs cancel. A near-zero mean cosine only rules out a single dominant cone, not bimodal/multimodal concentration.
- “Isotropy” is several inequivalent notions. Distinguish at least: spherical symmetry of the distribution; isotropic covariance ; uniformity of the normalized directions on the sphere; a flat covariance spectrum; and absence of a dominant mean direction. These do not coincide without extra assumptions.
Centering is essential. A high mean cosine often reflects a large mean rather than covariance anisotropy. It is better to report several quantities separately: the mean-direction norm , the spectral anisotropy of the centered data, and the angular distribution of normalized vectors — keeping the raw mean pairwise cosine as one specific empirical indicator, not the definition of (an)isotropy.
Intuition
Picture every activation vector as an arrow from the origin. In an isotropic space, the arrows point every which way, roughly evenly covering all directions — like the spines of a sea urchin. In an anisotropic space, almost all the arrows are bunched into a narrow ice-cream-cone-shaped wedge around some common direction, even though they can still have very different lengths and slightly different angles within that wedge.
Properties
-
A property of the whole representation space, not of any one vector’s meaning. Anisotropy is a statistic about how a population of vectors is directionally distributed — it says nothing on its own about whether any individual vector’s direction is semantically meaningful (contrast with Linear Direction, which is about one specific direction’s causal/semantic role).
-
Must be adjusted for before other geometric statistics are trusted. Ethayarajh (2019) shows the same-word self-similarity and same-sentence intra-sentence similarity statistics are both inflated by ambient anisotropy — a word’s representations can look “well contextualized” or “poorly contextualized” purely as an artifact of how anisotropic the whole layer is, unless the raw statistic has the layer’s own anisotropy baseline subtracted off first.
-
Empirically present and increasing with depth in every contextualizing model tested (ELMo, BERT, GPT-2). Ethayarajh (2019) finds all three models’ contextualized layers are anisotropic (the only isotropic layer is ELMo’s non-contextual, character-level input layer), generally worsening in upper layers — in GPT-2’s last layer, two randomly chosen words have on average almost perfect cosine similarity.
-
In tension with known benefits of isotropy for static embeddings. Arora et al. (2017) show isotropy aids theoretical self-normalization; Mu et al. (2018) show empirically that post-hoc isotropizing static embeddings (mean-subtraction) improves downstream performance — making the degree of anisotropy found in contextualized representations “surprising” against that prior expectation, not an obviously-benign byproduct.
-
A cone in the directional sense here is a statistical/global property of the whole point cloud, not the same object as Cone.
manifolds-cone’s polyhedral/circular cone catalogues a specific, causally-meaningful set of generator directions (e.g. a behavior-mediating cone spanned by a handful of steering vectors); anisotropy’s “narrow cone” is a global statistical description of an entire layer’s activation cloud, with no claim that the shared direction it clusters around is itself meaningful. -
Not actually diffuse across many dimensions — dominated by a handful of “rogue” outliers. Timkey & van Schijndel (2021) decompose the cosine-similarity anisotropy statistic per-dimension and find just 1-5 high-variance, far-from-origin dimensions account for nearly all of it (in the most extreme case, one XLNet dimension explains over 99%); via KL-divergence ablation they further show these cosine-dominant dimensions are largely not the dimensions the model’s own behavior actually depends on — a “striking mismatch” between what dominates the geometric statistic and what matters causally. This reframes anisotropy as substantially an artifact of a small number of outlier dimensions rather than a genuinely global directional bias, and motivates per-dimension standardization (z-scoring) as a correction before trusting any cosine-similarity-based geometric analysis — see
rogue-dimensions-obscure-anisotropy. -
IsoScore-family metrics extend beyond static contextualization studies to a training-dynamics setting, where forcibly restoring isotropy turns out to hurt rather than help. Lanza, Pereira, Miozzo, Angelats & Dini (2026) measure IsoScore and a generalized IsoEntropy (covariance-eigenvalue-spectrum uniformity) on penultimate-layer features of real ResNet-18 encoders trained via SupCon, Co2L, and two neural-collapse-inspired variants on CIFAR-10/100 under class-incremental continual learning, finding isotropy declines as more tasks accumulate but is not consistently correlated with accuracy across methods. A causal test — adding a differentiable isotropy regularizer to the training loss — raises IsoScore sharply (0.023 to 0.969 on one config) while collapsing accuracy (70.64% to 14.74%), showing that forcibly restoring isotropy in a continual-learning setting is actively harmful rather than a free fix, a cautionary causal counterpoint to isotropy-improving interventions found beneficial elsewhere. See
lanza-etal-2026-real-resnet18-feature-spaces-lose-isotropy-under-continual-learning-and-forcing-isotropy-back-up-via-a-regularizer-degrades-rather-than-helps-accuracy. -
The measurement instrument itself matters, and a rigorously axiomatized one gives a different picture again. Rudman et al. (2021)‘s IsoScore — built to be mean-agnostic, scale-invariant, rotation-invariant, and monotonically sensitive to how many dimensions are used, properties average-cosine-similarity and the “partition score” each provably lack — finds contextualized models are even more anisotropic than cosine-based estimates suggested (GPT/GPT-2 don’t uniformly occupy even one full dimension), yet directly overturns two conclusions the older metrics had supported: there is no significant trend of increasing isotropy in earlier layers, and mean-centering does not “recover” isotropy (isotropy here is a function of the covariance matrix alone, unaffected by the mean by construction) — see
isoscore-refines-anisotropy-picture. -
Substantially, though not entirely, a between-cluster effect. Cai, Huang, Bian & Church (2021) cluster each layer’s representations (k-means) and recompute cosine-similarity statistics within each cluster after center-shifting: inter-type cosine similarity is close to (isotropic) within clusters across every layer of BERT, DistilBERT, GPT, GPT-2, and ELMo, even though the unadjusted, whole-population statistic reads as strongly anisotropic — reframing (not overturning) Ethayarajh’s finding as real at the population level but substantially explained by distinct clusters occupying different regions of the space, rather than every individual vector pointing in a similar direction. The same paper separately finds a low-dimensional manifold structure (via Local Intrinsic Dimension) coexisting with this within-cluster directional isotropy — two distinct senses of “isotropy” that are both true at once, not in tension. See
clusters-explain-apparent-anisotropyand Intrinsic-dimension profile across depth‘smanifold-dimension-lower-than-ambient. -
Frequency-conditioned, and correctable without discarding information. Li et al. (2020) show BERT’s anisotropic word/sentence embedding space is not uniformly distorted but specifically frequency-biased — high-frequency words cluster near the origin, low-frequency words sit far away and disperse sparsely among their own neighbors, leaving low-probability-density “holes” a sentence embedding (an average of context embeddings) can land in. BERT-flow maps the embedding space to an isotropic Gaussian via an invertible, information-preserving normalizing flow, correcting this without the information loss of nulling top singular vectors — see
bert-flow-corrects-anisotropic-holes. -
A much simpler closed-form linear fix works nearly as well as a learned one. Huang et al. (2021) show whitening — an exact, trainable-parameter-free linear map computed directly from the embedding covariance’s eigendecomposition — consistently improves sentence-similarity performance across a wide range of pretrained models, without the training cost of a full normalizing flow, since matching just the first two empirical moments (mean and covariance) to an isotropic target already captures most of the practical benefit — see
whitening-improves-sentence-similarity. -
The modality gap is a mode-failure of the InfoNCE loss specifically at low temperature, not primarily an initialization/cone effect, and adding intra-modal negative pairs (xNCE) reduces the gap while matching or improving retrieval and zero-shot accuracy. Mager, Nassar & Hansen (2026) prove and empirically demonstrate this in a uni-modal control setting (two independently-initialized encoders under identical initialization conditions) with CLIP-style dual encoders, isolating the loss function’s own contribution to the gap from initialization dynamics — a third competing/complementary account of the modality gap’s cause alongside the cone-effect account and the loss-uniformity account already in this section. See
the-clip-modality-gap-is-an-infonce-low-temperature-mode-failure-fixed-by-adding-intra-modal-negatives. -
Under a subspace constraint (dimension collapse), the modality gap provably converges to the angle between two modality-specific hyperplanes, and closing that angle via a shared-space projection on real CLIP causally improves both the gap and zero-shot accuracy over prior baselines. Yi, Douady & Chen (2025) prove the gap converges to zero without a subspace constraint or under a cone constraint, but to the smallest angle between two hyperplanes once dimension collapse is imposed, then apply their own Shared Space Projection to real CLIP ViT-B/32 embeddings, closing the gap angle from 74.69 to 5.37 degrees on CIFAR-10 (vs. 7.02 for a translation baseline) while better preserving zero-shot retrieval accuracy (86.43 vs. 80.97) — a fourth account, the only one with a formal necessary-and-sufficient proof tying dimension collapse to a nonzero gap. See
the-clip-modality-gap-converges-to-the-angle-between-two-modality-specific-hyperplanes-under-dimension-collapse. -
A per-token geometric signature combining cross-layer covariance dimensionality and circular-variance anisotropy detects LLM hallucination better than prior uncertainty baselines. Srey et al. (2026) show their SIVR signature (log-pseudo-determinant of cross-layer hidden-state covariance plus circular variance of layer-wise normalized hidden vectors) reaches up to 89.5% AUC on TriviaQA for Llama-3.1-8B, beating prior baselines across seven QA/reasoning benchmarks. See
per-token-log-pseudo-determinant-of-cross-layer-covariance-and-circular-variance-of-hidden-states-detect-llm-hallucination-better-than-prior-uncertainty-baselines. -
The residual stream’s distributional skewness is measurably higher in later layers when an LLM favors contextual over parametric knowledge under a conflict. Zhao et al. (2024) quantify this via Kurtosis/Hoyer/Gini metrics from layers 20-30 in Llama3-8B and Llama2-7B, a directly measured geometric-shape difference tied to which knowledge source the model relies on. See
logistic-regression-probes-on-the-residual-stream-detect-parametric-vs-contextual-knowledge-conflict-and-predict-which-source-the-model-will-rely-on-before-generation-with-a-distinct-skewness-signature. -
The fraction of embedding variance held by the single dominant (“rogue”) dimension predicts whether cosine or rank-based similarity metrics win, across 19 text encoders, and causally projecting out dominant dimensions erases the effect only where it was anisotropic to begin with. Parupudi (2026) finds rank correlation 0.86 and linear correlation 0.95 between rogue-dimension variance share and the metric-improvement magnitude. See
the-fraction-of-embedding-variance-held-by-the-single-dominant-dimension-predicts-whether-cosine-or-rank-based-similarity-metrics-win-across-19-text-encoders-and-projecting-out-dominant-dims-erases-the-effect. -
Foundational: static word embeddings share a dominant mean plus a handful of top-PC directions, and removing them causally improves downstream tasks. Mu & Viswanath (2018) measure, via PCA/SVD on word2vec/GloVe-family embedding matrices across multiple languages, that a large shared mean vector plus a small number of top principal directions account for most of the variance — a foundational, pre-contextualized-model anisotropy measurement. Their “all-but-the-top” postprocessing (subtract the mean, project out the top directions) causally improves word similarity, analogy, categorization, STS, and text-classification benchmarks with no retraining, establishing both the measurement and the causal-correction pattern that later contextualized-model anisotropy work (Ethayarajh 2019 onward) builds on. See
mu-viswanath-2018-word-embeddings-share-a-dominant-mean-vector-plus-top-pc-anisotropy-and-removing-them-causally-improves-downstream-task-performance. -
Foundational: tied input-output embeddings in real trained NLG models degenerate into a narrow cone, with a training-dynamics explanation for why. Gao, He, Tan, Qin, Wang & Liu (2019) measure this cone directly in real trained language/translation models with weight-tied embeddings and explain it as a consequence of softmax normalization combined with weight tying, which pushes especially rare-word vectors toward a shared direction; their proposed cosine-similarity regularizer (MLE-CosReg, not “MMI”) measurably widens the embedding spread. Predates and matches the pattern later confirmed in contextualized (BERT-family) models. See
gao-etal-2019-tied-input-output-embeddings-in-real-trained-nlg-models-degenerate-into-a-narrow-anisotropic-cone-explained-by-softmax-weight-tying-dynamics. -
A mixture-of-experts router suppresses its own hidden states’ dominant shared direction under load-balancing loss, and hidden-state similarity predicts expert-usage similarity across five real production MoE models. Wang, Hayou & Nalisnick (2026) show real hidden states in five production MoE models — gpt-oss-20B, ERNIE-4.5-21B-A3B-Base, Qwen-3-30B, Ling-mini and Trinity-Mini-Base (Moonlight-16B is a comparison only) — develop a dominant shared direction (first right singular vector) whose retained energy grows with depth, and prove the auxiliary load-balancing loss forces the router to suppress this shared direction to maintain diverse outputs; separately, hidden-state cosine similarity tracks expert-usage similarity so closely that two different models’ solutions to the same question show ~60% expert overlap, comparable to the same model on two different questions. See
wang-etal-2026-real-pretrained-moe-routers-suppress-a-shared-dominant-hidden-state-direction-under-load-balancing-loss-and-hidden-state-cosine-similarity-predicts-expert-usage-similarity-across-five-production-models. -
Visual tokens fed into real trained MLLMs partition into discrete functional clusters, each collapsing to near-perfect cross-image cosine similarity. Fan, Tong, Zhao & Shen (2026) cluster LLaVA-1.5-7B’s visual tokens (norm threshold plus cosine similarity of cluster centroids) into “Sink,” “Dead,” and “Alive” categories (sink+dead ~40% of all tokens); sink-cluster centroids reach cross-image cosine similarity >0.99 (ViT stage) and >0.95 (LLM stage) — a near-total directional collapse for specific functional subgroups rather than the whole population, confirmed causally distinct via cluster-targeted ablation (removing sink/dead tokens leaves overall accuracy flat or improves a CV-centric sub-score). See
fan-etal-2026-visual-tokens-fed-into-real-trained-mllms-partition-into-sink-dead-and-alive-functional-clusters-with-near-perfect-cross-image-cosine-similarity-of-cluster-centroids. -
A bell-shaped, middle-layer-peaked anisotropy profile distinguishes decoder-only from encoder architectures. Razzhigaev, Mikhalchuk, Goncharova, Oseledets, Dimitrov & Kuznetsov (2024) compute an SVD-based anisotropy score layer-by-layer across a wide range of real pretrained encoders (BERT, RoBERTa, ALBERT) and decoders (OPT, Llama-2, GPT-2, GPT-J, Falcon, Bloom, Pythia, TinyLlama): “decoders showcase a unique bell-shaped structure, indicating that the middle layers tend to have a higher anisotropy concentration,” a depth-wise shape distinct from encoders’ own profile — purely observational, no causal intervention. See
razzhigaev-etal-2024-decoder-only-transformers-develop-a-bell-shaped-middle-layer-anisotropy-concentration-unlike-encoders-across-a-wide-range-of-real-pretrained-models. -
“Massive values” concentrate at consistent, cross-head dimension indices in real RoPE transformers’ query/key space, absent without RoPE. Jin, Mei, Xu, Sun, Tang, Du, Liu & Zhang (2025, ICML 2025) define a massive value as a per-(head, dimension) L2 norm exceeding 5x the mean, finding these cluster at nearly identical dimension indices across attention heads in real Llama-3-8B-Instruct, Gemma-2-9B-it, Qwen2.5-7B-Instruct and Llama-2-7B-chat, but not in non-RoPE GPT-2 or OPT. Ablating (mean/zero/min replacement) the massive-value dimensions during prefilling collapses contextual-knowledge tasks (e.g. GSM8K 76.9%→4.0% on Llama3-8B; passkey retrieval 100%→0%), and quantization schemes that preserve these dimensions (AWQ/SmoothQuant) retain performance that GPTQ (which does not) loses — a rogue/outlier-dimension-style concentration pattern with a strong causal signature, distinct in mechanism (tied specifically to rotary position embeddings) from the cosine-similarity-decomposition route of Timkey & van Schijndel above. See
jin-etal-2025-massive-values-concentrate-at-consistent-head-dim-indices-in-real-rope-transformers-q-k-space-and-ablating-or-quantizing-them-collapses-contextual-knowledge-tasks. -
Positional encodings causally anchor an index-based, rather than content-based, spatial-similarity organization in a real trained ViT. Mannes (2026) computes Spatial Similarity Distance Correlation (SSDC — the Spearman correlation between representation cosine similarity and negative spatial distance) across depth in a real ViT-S/16 trained from scratch on ImageNet-100, across five positional-encoding variants; all PE schemes shift token representations toward index-anchored rather than purely content-based spatial organization, and causal interventions (random permutation, positional-embedding magnitude scaling) show SSDC collapsing alongside rising fragility to distribution shift as positional-embedding magnitude is scaled down — a distance-dependent spatial-organization claim in a new (vision) modality, structurally analogous to Zuo, Guerzhoy & Guerzhoy’s distance-decaying adjacency pattern above. See
mannes-2026-positional-encodings-causally-anchor-an-index-based-rather-than-content-based-spatial-similarity-organization-in-a-real-trained-vit-and-removing-it-increases-fragility.
Exercises
Base
- If every vector in were identical (all equal to some fixed ), what would equal?
Solution
for every pair, so — the maximum possible anisotropy, consistent with a “cone” that has collapsed to a single ray.
- Why does Ethayarajh (2019) subtract a per-layer anisotropy baseline from self-similarity and intra-sentence similarity before interpreting them, rather than using the raw statistics directly?
Solution
Because a high raw self-similarity could mean either “this word’s representations are genuinely stable across contexts” or simply “this whole layer is so anisotropic that even two unrelated random words would already have high cosine similarity” — the two explanations are observationally identical unless the layer-wide baseline (average similarity between random word pairs) is subtracted off first, isolating the word-specific signal from the ambient directional bunching.
Middle
- Ethayarajh reports that GPT-2’s raw (non-anisotropy-adjusted) maximum explainable variance (MEV) is around 30% in middle layers, while the anisotropy-adjusted MEV stays under 5% in every layer of every model tested. Explain how both numbers can be true simultaneously without contradiction.
Solution
Raw MEV measures how much variance a word’s own representations’ first principal component explains, without accounting for the fact that a single shared direction (the anisotropic bulk direction) already explains a large fraction of variance across any random collection of vectors in that layer, word-specific or not. Subtracting the baseline MEV (computed the same way but over randomly sampled, unrelated word representations) isolates how much additional, word-specific variance-explaining power the first principal component has beyond what ambient anisotropy alone would already provide. GPT-2’s high raw MEV is therefore mostly attributable to its extreme anisotropy, not to genuine word-sense clustering — exactly the adjustment’s purpose.
-
The same “rogue dimension” phenomenon, extended to a new modality and given a quantified positional structure. Cheng, Zheng, Xie, Liao & Li (2026) show video diffusion transformers (Wan2.1, Wan2.2, CogVideoX) have their own massive-activation channels — a small, fixed set of dimensions taking magnitudes “often exceeding 50x the mean activation value,” consistent across layers, model scales, and text conditioning. Unlike the token-agnostic outlier dimensions of Timkey & van Schijndel’s LLM setting, these follow a quantified token-position hierarchy tied to the model’s own latent temporal chunking (first-frame > latent-frame-boundary > interior tokens, with the boundary-to-interior ratio declining toward 1 over the course of denoising). Rather than only measuring the outlier dimensions’ causal (ir)relevance via ablation, Structured Activation Steering (STAS) directly overwrites their values toward a scaled reference magnitude at the identified token positions, causally improving generation quality (VBench total score, all three models) — one of the only entries in this map where amplifying, rather than removing or standardizing, an anisotropy-driving outlier dimension is shown to be the behaviorally beneficial intervention, and a uniform (position-agnostic) version of the same edit is shown to hurt quality by way of contrast. See
cheng-etal-2026-massive-activation-channels-in-video-dits-follow-a-quantified-token-position-hierarchy-and-causally-steering-them-improves-video-quality. -
A distance-dependent sub-structure distinct from the population-level statistic. Zuo, Guerzhoy & Guerzhoy (2025) show that in causal-attention Transformers with no positional encoding, the self-cosine-similarity matrix isn’t just uniformly high (the global anisotropy story) — it decays specifically with token distance: nearby tokens are systematically more similar than far-apart ones (the “adjacency pattern”), quantified by a novel row-wise monotonicity statistic (the adjacency probability score) that jumps from ~0.5 at the raw token-embedding layer to ≥0.8 immediately after the first causal-attention layer, in both randomly initialized and trained 6-layer NanoGPT models on four synthetic position-sensitive tasks. This pattern is provably expected from the approximate orthogonality of high-dimensional embeddings plus the convex-combination structure of causal-attention outputs, and using cosine similarity to the last-position embedding as a linear-probe feature predicts token position better than the embedding’s own norm/variance (a competing account of how positional information is stored). See
zuo-etal-2025-causal-attention-without-positional-encodings-produces-a-distance-decaying-adjacency-pattern-in-the-self-cosine-similarity-matrix-quantified-by-a-novel-adjacency-probability-score. -
A state-space (Mamba) architecture shows anisotropy roughly an order of magnitude tighter than BERT’s, co-occurring with representational collapse, and a geometry-targeted causal fix fails. Wagh & Singh (2026) measure a mean pairwise cosine similarity of 0.9999 in frozen Mamba-130m (vs. BERT’s ~0.99), alongside a CoLA linear probe collapsing to majority-class prediction (MCC=0.000); a modified SSM recurrence forcing new state writes to be perpendicular to the current state (“orthogonal injection”) leaves the cosine similarity unchanged and worsens STS-B correlation — a clean negative causal result reported candidly. See
wagh-singh-2026-frozen-mamba-130m-activations-collapse-into-an-extremely-narrow-anisotropy-cone-10x-tighter-than-bert-and-a-geometry-targeted-orthogonal-injection-fails-to-widen-it. -
A cross-modal magnitude asymmetry, not the usual within-modality directional cone. Qi, Liu, Tang & Zhu (2025) measure real LLaVA-1.5-7B vision- and text-token embeddings on 5,000 COCO pairs, finding vision-token L2 norms (~10^1-10^3) dwarf text-token norms (~10^-1-10^0) by a measured factor of 10-100x, suppressing the LLM’s positional (RoPE) sensitivity and causing spatial-reasoning failures — a magnitude-scale asymmetry between modalities rather than a within-modality directional cone, but structurally analogous to the CLIP/contrastive modality gap already on this page. RMS-renormalizing vision embeddings to the text-embedding norm scale causally and predictably restores positional sensitivity, improving accuracy on five benchmarks (e.g. a synthetic 2D spatial task: 56.63%→64.80% combined with multilayer CLIP-feature extraction). See
llava-vision-token-embeddings-have-100x-larger-norms-than-text-suppressing-positional-sensitivity-and-normalization-restores-it.
Key papers
- Mu, Jiaqi & Viswanath, Pramod (2018). All-but-the-Top: Simple and Effective Postprocessing for Word Representations. ICLR. arXiv:1702.01417 — foundational static-embedding anisotropy measurement and causal all-but-the-top correction.
- Wagh, Bhagyashree & Singh, Akash (2026). Lost in State Space: Probing Frozen Mamba Representations. arXiv:2605.00253 — extreme SSM anisotropy and a negative orthogonal-injection causal test.
- Zuo, Guerzhoy & Guerzhoy (2025). Position Information Emerges in Causal Transformers Without Positional Encodings via Similarity of Nearby Embeddings. COLING 2025, arXiv:2501.00073 — the distance-decaying “adjacency pattern” and adjacency probability score.
- Ethayarajh (2019). How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings. EMNLP-IJCNLP, aclanthology D19-1006 — origin of the anisotropy-adjusted contextuality measures (self-similarity, intra-sentence similarity, maximum explainable variance) for contextualized LMs.
- Mimno & Thompson (2017). The Strange Geometry of Skip-gram with Negative Sampling. EMNLP — origin of the “narrow cone” anisotropy finding for static word embeddings, which Ethayarajh (2019) extends to contextualized models.
- Arora, Liang & Ma (2017). A Simple but Tough-to-Beat Baseline for Sentence Embeddings. ICLR — theoretical grounding for why isotropy corresponds to near-zero expected cosine similarity between random vectors.
- Timkey & van Schijndel (2021). All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality. EMNLP, 2021.emnlp-main.372, arXiv:2109.04404 — shows anisotropy is dominated by a handful of outlier dimensions largely irrelevant to model behavior; see Rogue-dimension analysis.
- Rudman, Gillman, Rayne & Eickhoff (2021). IsoScore: Measuring the Uniformity of Embedding Space Utilization. arXiv:2108.07344 — a rigorously axiomatized isotropy metric that overturns two prior cosine-based conclusions about layer depth and mean-centering; see IsoScore.
- Li, Zhou, He, Wang, Yang & Li (2020). On the Sentence Embeddings from Pre-trained Language Models. EMNLP, 2020.emnlp-main.733, arXiv:2011.05864 — traces anisotropy to frequency-conditioned “holes” in the embedding space and corrects them via an invertible normalizing flow; see Normalizing-flow calibration (BERT-flow).
- Huang, Tang, Zhong, Lu, Shou, Gong, Jiang & Duan (2021). WhiteningBERT: An Easy Unsupervised Sentence Embedding Approach. Findings of EMNLP, 2021.findings-emnlp.23 — shows a simple closed-form linear whitening transform corrects anisotropy nearly as effectively as a learned normalizing flow, across a wide range of pretrained models; see Whitening transformation.
- Liang, Zhang, Kwon, Yeung & Zou (2022). Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning. NeurIPS 2022, arXiv:2203.02053 — extends the narrow-cone finding to two-encoder contrastive (image-text) models, proves an initialization-driven cause via two theorems, and causally manipulates the resulting inter-modality displacement vector; see
clip-modality-gap-is-a-provable-cone-effect-plus-causally-tunable-offset.