Transcoders

methods / Dictionary Learning / Transcoders

Techniqueadvanced

A sparse dictionary trained to approximate an entire MLP sublayer's input-to-output function (rather than reconstructing a single site's own activations, as an SAE does), whose ReLU-gated decoder-vector sum turns weights-based circuit attribution through the MLP into an exact linear factorization of input-dependent (feature activation) and input-invariant (decoder-encoder dot product) terms.

Used in (3 observations)

Transcoder-decoded planning directions causally control future tokensmeasured

structure: Linear Direction · models: Qwen3-0.6B, Qwen3-1.7B, Qwen3-4B, Qwen3-8B, Qwen3-14B · paper: Latent Planning Emerges with Scale

Timestep-conditioned transcoder directions give exact FLUX attribution graphsmeasured

structure: Linear Direction · models: FLUX.1 [schnell] · paper: DifFRACT: Diffusion Feature Reconstruction and Attribution for Circuit Tracing

Transcoders linearize MLP circuit attribution and match SAE qualitymeasured

structure: Linear Direction · models: GPT-2 Small, Pythia-410M, Pythia-1.4B · paper: Transcoders Find Interpretable LLM Feature Circuits