Dimension-level activation outliers, measured as the ratio gamma of mean norm to per-token variation, predict sparse-autoencoder feature death across 454 model-layer combinations spanning language, vision, protein and genomic models

- Simon, Adams and Zou report that dimension-level activation outliers (dimensions whose mean magnitude is large relative to per-token variation) shift SAE pre-activations at initialization according to each feature's alignment with the activation mean, so features anti-aligned with the mean never fire; the severity ratio gamma = ||mu||/||sigma|| predicts initial death rates (Spearman 0.89 dead-by-TopK, 0.82 dead-by-ReLU) across 454 model-layer combinations, and mean-centering eliminates the outlier-induced death at the representative layer of every model tested except Evo1, where 73% of features stay dead at initialization because 4 of 4096 principal components capture 99% of the variance (a separate low-rank cause fixed by PCA whitening). The 454 layers come from 19 models (Table S1: language, vision, protein and genomic; AlphaFold3 and the GPT-2 contrast of near-zero versus about 72% dead), trained with TopK SAEs on 10M-token extractions; only the models that match an existing model entry exactly are tagged (GPT-2 size is not stated in the paper). The node choice (mean-dominated, high-mean/low-variance dimensions under anisotropy) is tentative: the paper never uses the word anisotropy. [simon-etal-2026-on-the-relationship-between-activation-outliers-and-feature-death-in-s]