SmolLM2

Hugging Face

Structures found in this family (3)

Linear Subspace, Linear Direction, Tangent-Aligned Anisotropy Hypothesis

By model (2)

SmolLM2-1.7B · 1.7B

Linear Subspace, Linear Direction, Tangent-Aligned Anisotropy Hypothesis

SmolLM2-360M · 360M

Tangent-Aligned Anisotropy Hypothesis, Linear Subspace

Observations (3)

Around successful trajectories the pullback Fisher operator on hidden-state space has a thin anisotropic spectrum that separates output-sensitive residual-stream directions from approximately prediction-null directions across ten decoder-only LLMs

Concept directions anti-concentrate in the unembedding's low-variance spectral tail

Anisotropy from frequency-blind sampling and self-reinforcing tangent gradients

Papers

Predictive Geometry of Hidden Trajectories in Transformers (2026), Concepts Whisper While Syntax Shouts: Spectral Anti-Concentration and the Dual Geometry of Transformer Representations (2026), Revisiting Anisotropy in Language Transformers: The Geometry of Learning Dynamics (2026)