A single token's residual linearly decodes an entire maze

Ivanitskiy et al. train small (<10M) GPT transformers from scratch on synthetic maze token sequences with one-hot orthogonal input tokens [ivanitskiy-etal-2023] Linear probes on a single fixed token's residual stream reconstruct the entire maze's wall structure (>90% wall accuracy, peaking at layer 2), a striking single-site compression [ivanitskiy-etal-2023] Learned embedding vectors develop emergent spatial structure, with Manhattan distance correlating with embedding L1 distance at short range despite orthogonal inputs [ivanitskiy-etal-2023] Candidate "Adjacency Heads" attend to path-adjacent tokens in-context, correlational evidence not causally ablated in the paper [ivanitskiy-etal-2023] Sharp improvements in linear maze decodability coincide with sharp task-generalization gains, a grokking-like transition [ivanitskiy-etal-2023]