Unsupervised orthogonal subspace partitions of the residual stream localize circuit variables

Neighbor distance minimization (NDM) learns a rotation and partition of residual-stream activations into non-basis-aligned subspaces using only activations [huang-hahn-2025] On a 5-test GPT-2 Small suite built from the IOI and greater-than circuits, subspace patching effects have average Gini 0.71 for NDM versus 0.21 for identity or random rotations and 0.56 for SAE-feature clustering [huang-hahn-2025] In Qwen2.5-1.5B (layer 11, mid) NDM finds a 256-dim subspace that only mediates parametric-knowledge routing and a 64-dim subspace that only mediates context-knowledge routing under subspace patching [huang-hahn-2025] In Gemma-2-2B (layer 9, mid) some NDM subspaces have strong patching effect under one kind of counterfactual input but negligible effect under the other [huang-hahn-2025] The paper states that interpreting subspace directions by cosine similarity relies on the Linear Representation Hypothesis, which could be untrue in some cases [huang-hahn-2025]