Transcoders linearize MLP circuit attribution and match SAE quality

measured in 1 paper

Dunefsky et al. introduce transcoders: SAE-shaped networks trained to reconstruct an MLP sublayer's output from its input rather than the MLP's own activations [dunefsky-etal-2024-transcoders] Circuit attribution then factorizes exactly into an input-dependent activation term times a fixed decoder-encoder dot product, needing no gradient/Taylor linearization of the MLP [dunefsky-etal-2024-transcoders] On GPT2-small, Pythia-410M, and Pythia-1.4B, transcoders sit on reconstruction/sparsity Pareto frontiers equal to or better than SAEs, with the gap widening at larger scale [dunefsky-etal-2024-transcoders] Applying the exact-factorization method to GPT2-small's greater-than circuit surfaces a finer feature-level account and isolates one anomalous flat direct-logit-attribution feature [dunefsky-etal-2024-transcoders]

Structure

Linear Direction

Context

sparse dictionary reconstructing an MLP's output from its input, not the MLP's own activations, exact (non-Taylor) linear factorization of weights-based circuit attribution into input-dependent and input-invariant terms, reconstruction/sparsity Pareto frontier at least matching SAEs, gap widening with model size, finer-grained feature-level reverse-engineering of a previously neuron-level-only circuit

Confirmed in models

GPT-2 Small · GPT-2, Pythia-410M · Pythia, Pythia-1.4B · Pythia

Method

Transcoders

Papers

Transcoders Find Interpretable LLM Feature Circuits — Dunefsky, Jacob, Chlenski, Philippe, Nanda, Neel2024 · arXiv:2406.11944