LLaMA-3.1-8B linearly encodes grid position; action-correctness units are causal

measured in 1 paper

Martorell has LLaMA-3 models (1B to 90B, with the 11B/90B being Vision checkpoints run text-only) navigate a 5x5 grid described in 6 spatial text encodings, with Cartesian encodings and larger models best (90B reaches 98% on JSON) [martorell-2025-grid-world-spatial-models] All linear-probing and ablation is done on LLaMA-3.1-8B only; per-layer regression probes decode a 50-dim agent/goal one-hot vector significantly above a permutation null at every layer and encoding [martorell-2025-grid-world-spatial-models] Univariate analysis finds unit sets correlating across all 6 encodings with agent x-position (448), y-position (258), border status (373) and action correctness (286), the last also elevated on unrelated spatial-reasoning prompts [martorell-2025-grid-world-spatial-models] Ablating the 286 action-correctness units drops JSON navigation success 59%->55% while ablating position units leaves it unchanged, a within-paper causal dissociation [martorell-2025-grid-world-spatial-models]

Structure

Linear Subspace

Context

50-dim one-hot position probe (linear regression, R^2, permutation-test null), format-invariant unit sets identified via per-unit significance testing, action-correctness units generalize to unrelated spatial-reasoning prompts, dissociable ablation effect (action-correctness units causal, position units not), representation-format effect on navigation success/efficiency (Cartesian > Topographic/Textual)

Confirmed in models

Llama-3.2-1B-Instruct · Llama, Llama-3.2-3B-Instruct · Llama, Llama-3.1-8B-Instruct · Llama, Llama-3.1-70B-Instruct · Llama

Method

Linear probing, Causal interventions (steering)

Papers

From Text to Space: Mapping Abstract Spatial Models in LLMs during a Grid-World Navigation Task — Martorell, Nicolas2025 · arXiv:2502.16690