MATH · IN · MODELS
methods / Theoretical / Analytical / Gradient–Tangent Alignment Test

Gradient–Tangent Alignment Test

Techniqueadvanced

Fits a low-rank activation-derived tangent-subspace proxy once from pooled early-checkpoint activations, then tests whether true backpropagated gradients concentrate in it more than in a matched-rank random subspace, via a gradient-energy ratio (with a Monte Carlo null) and an IsoScore*-removal test (does ablating the proxy recover more isotropy than ablating a random subspace).

Used in (1 observation)

structure: Tangent-Aligned Anisotropy Hypothesis, Linear Subspace · models: EuroBERT-210m, EuroBERT-610m, Pythia-160M, Pythia-410M, Pythia-1B, Pythia-1.4B, SmolLM2-360M, SmolLM2-1.7B · paper: Revisiting Anisotropy in Language Transformers: The Geometry of Learning Dynamics