A Top-k SAE on sentence embeddings gives near-orthogonal, steerable latents

Shin & Kim train a Top-k SAE (12x expansion, k=32) on real E5-large-v2 sentence embeddings over ~3.8M WikiText-103 sentences [shin-kim-2026-aligning-sentence-embeddings-saes] Decoder directions are near-orthogonal (mean pairwise cosine 0.0408) at 92.6% explained variance [shin-kim-2026-aligning-sentence-embeddings-saes] Zeroing specific interpretable latents (auto-labeled via GPT-4o-mini) measurably re-ranks retrieval results (an original top-1 result dropping to 9th place) [shin-kim-2026-aligning-sentence-embeddings-saes] The authors flag it as a single-backbone, English-only, proof-of-concept study [shin-kim-2026-aligning-sentence-embeddings-saes]