Around successful trajectories the pullback Fisher operator on hidden-state space has a thin anisotropic spectrum that separates output-sensitive residual-stream directions from approximately prediction-null directions across ten decoder-only LLMs

- Mudarisov, Burtsev, Petrova and State show that second-order geometry of layerwise loss-to-go functions near successful validation trajectories is governed by a pullback Fisher operator on hidden-state space, whose spectrum identifies output-sensitive and approximately prediction-null directions (a local observable subspace of the residual stream, thin and anisotropic), estimated with matrix-free Jacobian-vector products on ten decoder-only LLMs of about 1-9B parameters (Table 2 checkpoints) over WikiText, OpenWebText and FineWeb. The observable subspace is local (defined at each reference hidden state from the top eigendirections of the pullback Fisher operator, not a single global subspace): equal-norm perturbations in observable directions change the output KL by orders of magnitude more than null ones at almost every layer, and the effective observable rank (participation ratio) grows with depth but stays far below the hidden dimension. Geometry-guided rank allocation helps at moderate spectral thresholds for several models (e.g. Gemma-2-2B, Qwen2.5-3B, SmolLM2-1.7B, OLMo-3-7B) with a smaller advantage for others, and the token score is competitive with attention for pruning. The phi-3-mini entry is the instruct checkpoint used (microsoft/Phi-3-mini-4k-instruct); the other nine are base checkpoints (Table 2). [mudarisov-etal-2026-predictive-geometry-of-hidden-trajectories-in-transformers]