Gated SAEs eliminate shrinkage and halve the required L0

measured in 1 paper

Rajamanoharan et al. show a standard L1 ReLU SAE conflates deciding which directions are active with estimating their magnitudes, causing shrinkage (systematic underestimation) [rajamanoharan-etal-2024-gated-saes] The Gated SAE splits these into weight-tied gate and magnitude sub-networks, applying the L1 penalty only to the gate's pre-activations [rajamanoharan-etal-2024-gated-saes] Measured via relative-reconstruction-bias gamma, Gated SAEs reach gamma~1 while baseline SAEs show gamma<1 (0.72 at a Gemma-7B site) [rajamanoharan-etal-2024-gated-saes] Gated SAEs need roughly half the L0 for equal loss recovery across GELU-1L, Pythia-2.8B, and Gemma-7B, with at-least-equal interpretability [rajamanoharan-etal-2024-gated-saes] Under tied weights the construction is provably equivalent to a JumpReLU encoder [rajamanoharan-etal-2024-gated-saes]

Structure

Linear Direction

Context

weight-tied gate/magnitude split isolating the L1 penalty's effect, shrinkage (systematic magnitude underestimation) quantified via a relative-bias metric gamma, roughly half the active-feature count needed for equivalent reconstruction fidelity, algebraic equivalence to a JumpReLU encoder under tied weights

Confirmed in models

GELU-1L · GELU Toy Transformer, Pythia-2.8B · Pythia, Gemma-7B · Gemma

Method

Gated Sparse Autoencoders

Papers

Improving Dictionary Learning with Gated Sparse Autoencoders — Rajamanoharan, Senthooran, Conmy, Arthur, Smith, Lewis, Lieberum, Tom, Varma, Vikrant, Kramár, János, Shah, Rohin, Nanda, Neel2024 · arXiv:2404.16014