Participation-ratio spectral signal (per-head effective rank over training)

methods / Theoretical / Analytical / Participation-ratio spectral signal (per-head effective rank over training)

Participation-ratio spectral signal (per-head effective rank over training)

Techniqueadvanced

Computes the participation ratio (an effective-rank measure derived from the entropy of the squared singular-value distribution) of each attention head's per-token output activation matrix, integrated over training, to identify which heads are doing specialized computation without any behavioral labels.

Used in (2 observations)

Toy MLP's pullback metric localizes at class boundariesmeasured

structure: Decision boundary (as a codimension-1 hypersurface) · models: Small tanh MLP (AND/OR/XOR on synthetic toroidal/planar input manifolds, rich vs. lazy regimes) · paper: Emergent Riemannian Geometry over Learning Discrete Computations on Continuous Manifolds

A participation-ratio spectral signal finds attention circuits label-free across scalemeasured

structure: Intrinsic-dimension profile across depth · models: TS-51M (custom 8-layer x 512d x 16-head transformer, TinyStories, 6 pretraining seeds), nanoGPT-style GPT-2 124M (FineWeb-10B), Pythia-160M, Pythia-410M, Pythia-1B, OLMo-1B, OLMoE-1B-7B · paper: Spectral Probe-Circuits: A Three-Step Recipe for Identifying Attention-Head Circuits in Pretrained Transformers