TopK SAEs give clean scaling laws and mitigate dead latents

Gao et al. (OpenAI) replace an SAE's ReLU+L1 objective with a hard TopK activation trained on reconstruction alone, removing L1's shrinkage bias and fixing L0=k per token [gao-etal-2024-scaling-saes] An auxiliary loss reconstructing residual error from only currently-dead latents holds the dead-latent fraction to ~7% even in a 16M-latent autoencoder [gao-etal-2024-scaling-saes] Trained on GPT-4 activations for 40B tokens, a 16M-latent SAE costs only ~10% of GPT-4 pretraining-equivalent loss when spliced in, and reconstruction loss follows a clean joint scaling law in size and sparsity [gao-etal-2024-scaling-saes] Three feature-quality metrics (probe recovery, N2G explainability, ablation sparsity) all improve with size, and TopK SAEs beat ReLU baselines on every one [gao-etal-2024-scaling-saes]