Matryoshka Sparse Autoencoders

Trains one shared encoder/decoder pair but forces multiple nested prefixes of the dictionary (first $m_1<m_2<\dots<m_n$ latents) to each independently reconstruct the input via their own reconstruction loss term, so the smallest prefix is forced to capture general concepts on its own rather than letting sparsity pressure hand those concepts off to more specific latents that only exist in larger prefixes — directly targeting the feature-absorption failure mode of scaling up a single dictionary.