JumpReLU Sparse Autoencoders

methods / Dictionary Learning / JumpReLU Sparse Autoencoders

Techniqueadvanced

Replaces a standard SAE's ReLU activation with JumpReLU$_\theta(z)=z\odot H(z-\theta)$ — an elementwise, per-feature learnable threshold gate (Heaviside step function) — trained via straight-through estimators on a direct L0-sparsity penalty, allowing a variable number of active features per token (unlike TopK) while giving a Pareto-improved reconstruction/sparsity tradeoff over plain ReLU SAEs.

Used in (2 observations)

JumpReLU SAEs optimize L0 directly and match or beat Gated and TopKmeasured

structure: Linear Direction · models: Gemma-2-9B · paper: Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders

Gemma Scope releases 2,000+ JumpReLU SAEs; feature splitting persists with widthmeasured

structure: Linear Direction · models: Gemma-2-2B, Gemma-2-9B, Gemma-2-27B, Gemma-2-9B-it · paper: Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2