A GPT-2 trained on grid navigation forms an orthogonal xy coordinate map

Baumgartner et al. train GPT-2 small from scratch on three grid-navigation paradigms on a 4x4 grid [baumgartner-etal-2025-cognitive-maps-in-language-models] For the foraging task, PCA on node-token hidden states aligns the top-2 PCs at layer 7 with the grid x/y axes, and a linear probe recovers (x,y) at R^2=0.93 (layer 8) with near-orthogonal basis vectors (cos ~ -0.04) [baumgartner-etal-2025-cognitive-maps-in-language-models] Layer-by-layer ablation shows a sharp phase transition: loop-completion accuracy is near-zero when ablating before layer 8 but jumps to 100% at layer 8+, so the internal map becomes causally self-sufficient there [baumgartner-etal-2025-cognitive-maps-in-language-models] Goal-directed shortest-path variants show no orthogonal coordinate structure and no phase transition, a task-dependent dissociation [baumgartner-etal-2025-cognitive-maps-in-language-models]