Trains a sparse dictionary-learning autoencoder on model activations to recover a large set of candidate feature directions with no supervision; dictionary elements can be clustered to surface multi-dimensional features.

Replaces a standard SAE's ReLU-plus-L1-penalty with a hard TopK activation that keeps exactly the k largest pre-activations and zeroes the rest, fixing sparsity directly (removing the L1-shrinkage bias) and enabling a dead-latent-revival auxiliary loss (AuxK) that keeps dead-latent fraction low even at 16-million-latent scale, yielding clean joint scaling laws in autoencoder size and sparsity.

Relaxes TopK SAE's fixed per-sample active-latent count to a per-batch budget — keeping the top n×k activations across a batch of n samples rather than exactly k per sample — letting easy samples use fewer latents and hard samples use more, then restoring single-sample inference via a threshold estimated from the average minimum-positive-activation across batches.

A sparse dictionary-learning method — a generalization of SAEs — trained jointly across layers or models to recover shared, causally-relevant feature directions that persist across sites rather than one dictionary per site.

A sparse dictionary trained to approximate an entire MLP sublayer's input-to-output function (rather than reconstructing a single site's own activations, as an SAE does), whose ReLU-gated decoder-vector sum turns weights-based circuit attribution through the MLP into an exact linear factorization of input-dependent (feature activation) and input-invariant (decoder-encoder dot product) terms.

Splits an SAE's encoder into two weight-tied sub-networks — a gate deciding which features are active and a magnitude estimator sizing them — applying the L1 sparsity penalty only to the gate's pre-activations so magnitude estimation is no longer biased toward zero, provably equivalent to a single-layer JumpReLU encoder once the two sub-networks' weights are tied.

Replaces a standard SAE's ReLU activation with JumpReLU$_\theta(z)=z\odot H(z-\theta)$ — an elementwise, per-feature learnable threshold gate (Heaviside step function) — trained via straight-through estimators on a direct L0-sparsity penalty, allowing a variable number of active features per token (unlike TopK) while giving a Pareto-improved reconstruction/sparsity tradeoff over plain ReLU SAEs.

Trains one shared encoder/decoder pair but forces multiple nested prefixes of the dictionary (first $m_1<m_2<\dots<m_n$ latents) to each independently reconstruct the input via their own reconstruction loss term, so the smallest prefix is forced to capture general concepts on its own rather than letting sparsity pressure hand those concepts off to more specific latents that only exist in larger prefixes — directly targeting the feature-absorption failure mode of scaling up a single dictionary.

Builds a pairwise affinity matrix between dictionary atoms from co-occurrence statistics (e.g. Jaccard, Dice, phi coefficient) and runs spectral clustering on it, partitioning atoms by functional relatedness with no positional information used — the partition is then checked post hoc against the atoms' actual geometric positions.

Whitens (PCA-decorrelates) an embedding matrix, then applies Independent Component Analysis to find a privileged, non-orthogonal rotation of the whitened axes under which individual coordinates are maximally statistically independent and sparse — recovering interpretable, axis-aligned semantic components that PCA's rotation-free variance-maximizing axes do not.

Fits a pairwise Ising (binary graphical) model to a sparse dictionary's atom activations to recover the direct, conditional-independence-adjusted couplings between atoms — then groups atoms into candidate manifolds by their coupling sign/strength, rather than by decoder-direction similarity.

Groups an already-trained sparse dictionary's decoder atoms into candidate low-rank subspaces by fitting a mixture of linear subspaces (rather than point clusters) to the atom set, surfacing groups of latents that jointly span a shared direction set.

Classical (pre-neural-SAE) k-SVD-style sparse dictionary learning applied directly to a trained word-embedding matrix, recovering a small set of overcomplete 'atoms of discourse' such that each word vector is approximately a sparse linear combination of a few atoms.