MATH · IN · MODELS

Trajectory-subspace curvature perturbations causally modulate next-token entropy

measured in 1 paper

King et al. define contextual curvature as a 3-token backward average of the angle between consecutive residual-stream displacement vectors, extending the Hosseini-Fedorenko straightening measure [king-etal-2026-representational-curvature-modulates-behavioral-uncertainty] Across GPT-2 XL and Pythia-2.8B, contextual curvature predicts next-token entropy (Pearson r peaking ~0.15), rising across layers to a peak near the middle-layer curvature minimum [king-etal-2026-representational-curvature-modulates-behavioral-uncertainty] In Pythia's training the coupling is absent at 0-0.07% of a 300B-token run and emerges sharply around 0.7% [king-etal-2026-representational-curvature-modulates-behavioral-uncertainty] Only perturbations restricted to the recent-displacement subspace or its 2D plane causally move entropy; random, random-subspace, activation-PCA, and full-space perturbations of matched magnitude do not [king-etal-2026-representational-curvature-modulates-behavioral-uncertainty] Training a 12-layer model from scratch with a curvature-regularizing auxiliary loss lowers token-level entropy without degrading validation loss [king-etal-2026-representational-curvature-modulates-behavioral-uncertainty]

Context

contextual curvature (windowed average of adjacent-displacement-vector angle), curvature-entropy correlation rising across layers, peaking near middle-layer curvature minimum, curvature-entropy coupling emerging early in training (around 0.7% of a 300B-token run), trajectory-aligned perturbations (displacement-subspace, local plane) causally move entropy, unlike random/PCA/full-space controls, curvature-regularizing auxiliary training loss reduces token-level entropy without degrading validation loss

Papers

Representational Curvature Modulates Behavioral Uncertainty in Large Language Models — King, Jack, Fedorenko, Evelina, Hosseini, Eghbal A.2026 · arXiv:2604.23985