SAE activation-entropy effective-features count mirrors intrinsic-dimensionality studies

Bereska et al. define effective-features F = exp(Shannon entropy of SAE feature-activation magnitudes) and a superposition ratio psi = F/N as an effective-dimensionality measure of a representation [bereska-etal-2025-superposition-as-lossy-compression] Computed layer-by-layer on real pretrained Pythia-70M activations, the effective-features profile mirrors prior intrinsic-dimensionality studies of the same model [bereska-etal-2025-superposition-as-lossy-compression] The same metric captures a sharp feature-consolidation transition during grokking [bereska-etal-2025-superposition-as-lossy-compression] Dropout systematically reduces the number of effective features [bereska-etal-2025-superposition-as-lossy-compression]