Statement

Assume token embeddings concentrate locally near a $C^3$ manifold $\mathcal{M}$ (a local/stratified manifold hypothesis, not a single global smooth manifold), and that samples for a token are drawn with an ambient radial density $g(t)$, $t=\|y-\mu\|$, that is strictly decreasing — high-frequency tokens concentrate at small $t$ around their idealized centroid $\mu$, rare tokens remain more diffuse. Two coupled consequences follow:

Sampling-side (curvature attenuation). As radial concentration increases (a moment ratio $\eta_g \to 0$), the marginal directional distribution $p(u)$ over tangent directions $u$ becomes approximately uniform, and the curvature-driven directional bias that would otherwise favor high-(Ricci- and extrinsic-)curvature directions vanishes from what is empirically observable — curvature information is suppressed, not enhanced, by frequency-driven concentration.

Gradient-side (tangent dominance). Decomposing a centered activation $x_c=v+n$ into tangent ($v\in T_\mu\mathcal{M}$, $\|v\|=t$) and normal ($n\in N_\mu\mathcal{M}$, $\|n\|=O(t^2)$) components, the ratio of normal-to-tangent gradient magnitudes for any linear layer scales as $O(t)$: for small $t$ (frequency-concentrated tokens), weight updates along curvature-bearing normal directions are parametrically suppressed relative to tangent updates. This compounds through a feedback loop — tangent-aligned weight updates make forward activations more tangent-dominated, which biases future gradients further toward tangent directions — amplified by attention’s bilinear score expansion (the $v^\top M v$ term dominates the $O(t^3)$ cross-term and $O(t^4)$ normal-normal term) and propagated depth-wise by residual connections, which copy normal components forward unamplified while incrementally reinforcing tangent ones.

Intuition

Most accounts of anisotropy point to a static cause — weight tying, likelihood maximization, embedding norm shrinkage — and describe a narrow cone as a finished fact about trained representations. This hypothesis instead offers a dynamical, geometric mechanism: high-frequency tokens are sampled so tightly around their “ideal” spot that the curvature of the underlying semantic manifold becomes statistically invisible, and because gradients scale with how far a sample wanders from that spot, the training signal itself inherits and amplifies this blindness — normal (curvature) directions are not just under-sampled, they are actively starved of gradient signal, and the starvation gets worse as training proceeds and tangent directions come to dominate forward activations.

Properties

• Two independent, complementary mechanisms. The sampling-side curvature-attenuation argument (Corollary 2.2 → Eq. 4) concerns what is statistically visible in a sample of activations; the gradient-side tangent-dominance argument (Proposition 2.3) concerns what training reinforces in the weights. Either could in principle hold without the other, but both point the same direction and are empirically tested together.
• Self-reinforcing, not static. Unlike accounts that treat anisotropy as a fixed property of an objective function, this hypothesis is explicitly a feedback loop: tangent bias in weights → more tangent-dominated activations → more tangent-biased future gradients — predicting anisotropy should emerge and compound over training, not appear instantaneously.
• Predicts anisotropy is imprinted early, not throughout training. Because the mechanism depends on a not-yet-converged, actively updating model, and prior work shows representations stabilize early, the hypothesis predicts the tangent bias — and hence anisotropy’s origin — is strongest in early training, with late training carrying it forward under smaller updates rather than originating it.
• Predicts weaker anisotropy where frequency concentration is weaker. Models whose training explicitly counteracts frequency skew (e.g. multilingual data-balancing) should show an attenuated, not absent, version of the same mechanism — a specific, falsifiable scaling prediction rather than a binary present/absent claim.
• Reframes anisotropy as possibly adaptive. Rather than treating a narrow cone purely as a pathology, the hypothesis is compatible with anisotropy functioning as an implicit dimensionality reduction — shifting an overparameterized model toward an effectively lower-dimensional, more generalizable regime — a normative reframing, not just a mechanistic one.

Exercises

Base

1. Given $\|v\|=t$ and $\|n\|=O(t^2)$, and that gradient magnitude along a subspace scales with $\sqrt{\mathbb{E}[\|\cdot\|^2]}$ (Cauchy-Schwarz), reproduce the $O(t)$ scaling of the normal-to-tangent gradient ratio in Eq. (11).

2. Two token types have concentration radii $t_1 \ll t_2$ (type 1 is high-frequency, type 2 is low-frequency). Which type’s normal-direction (curvature) gradient signal is more suppressed relative to its own tangent signal?

Middle

3. Explain concretely why “tangent-aligned weight updates” causing “more tangent-dominated future activations” is a feedback loop and not merely a repeated application of the same one-time bias — i.e. why iterating the mechanism should make the bias worse over training steps, not just repeat it at constant strength.

4. The paper reports that encoder models (EuroBERT), which explicitly balance training data across languages, show weaker tangent-alignment effects than decoder models. Using the hypothesis’s core mechanism, explain why data-balancing should have this effect without assuming anything about encoder-vs-decoder architecture per se.

Key papers

• Bernas, Jourdan, Poché & Hudelot (2026). Revisiting Anisotropy in Language Transformers: The Geometry of Learning Dynamics. arXiv:2604.08764 — origin of this hypothesis; see Gradient–Tangent Alignment Test for the empirical validation methodology.
• Robinson, Dean, et al. (2025a;b) — cited by Bernas et al. as the source of the local/stratified-manifold view of token space this hypothesis builds on (token subspaces failing global smooth-manifold and even fiber-bundle models); not yet a full paper entry in this corpus.
• Rudman & Eickhoff (2024) — origin of IsoScore*, the isotropy metric used in Gradient–Tangent Alignment Test‘s removal test; not yet a full paper entry in this corpus.