Attention builds a shared world-state coordinate system; the MLP specializes it

Brenner et al. train one decoder-only transformer (L=4, H=4, d_model=128, HookedTransformer) from scratch, six times, on six 2D-lattice random walkers defined by two endpoints {(0,0),(4,0)} and three horizons {20,200,1000} [brenner-etal-2026] Held-out affine probes show the residual stream after the first attention block linearly encodes the full geometric sufficient statistic [x/t, y/t, log t] (R^2=0.84-0.99), while the post-MLP representation encodes the smaller task-specific next-step statistic [brenner-etal-2026] Linear CKA between post-attention activations across different walkers is high (0.95-0.97), so attention builds one shared coordinate system, while post-MLP cross-walker CKA drops sharply, so the MLP specializes it [brenner-etal-2026] The first attention block builds the statistic via uniform causal averaging of value-writes, with a near-additive token/position decomposition and antipodal move codes [brenner-etal-2026] Positions sharing an identical next-step distribution remain linearly distinguishable via their longer-horizon futures, evidence the network keeps more than the immediate objective needs [brenner-etal-2026]