SAEs dilute continuous manifolds rather than compactly capturing them

Bhalla et al. formalize manifold capture (a small fixed group of decoder atoms spanning the manifold, consistently reselected by the encoder) and identify three regimes: compact capture, tiling/shattering, and intermediate dilution [bhalla-etal-2026] Training five SAE architectures on Llama-3.1-8B layer-19 activations, variance explained by a restricted atom group plateaus well beyond each manifold's ambient dimension, so none achieve compact capture [bhalla-etal-2026] Features behave like overlapping population-code tuning curves that redundantly tile the manifold, placing all tested SAEs in the dilution regime [bhalla-etal-2026] Ising-coactivation analysis on SAE codes recovers known manifolds (temperature, colors, political bias) unsupervised and surfaces a new epistemic-uncertainty manifold [bhalla-etal-2026]