Spatial-relation vectors form an orthogonal antipodal composable basis

measured in 1 paper

- In Llama-3.2-3B-Instruct, spatial-relation vectors form a low-dimensional basis where inverse relations are antipodal (w_below ≈ -w_above): Above/Below cos 0.978, Left/Right 0.997, Front/Behind 0.995. [tehenan-etal-2025-spatial-world-models] - Different axes are near-orthogonal (cross-axis cosine ~0), and multi-relation directions equal the vector sum of their components (2D e.g. above+left mean cos 0.993). [tehenan-etal-2025-spatial-world-models] - 3D compositions are looser, spanning cosine 0.82-1.00 (angular deviation 3.9-34.4 degrees), e.g. above+right cos 0.88 (~28 degrees) and left+behind ~0.82 (34 degrees). [tehenan-etal-2025-spatial-world-models] - Steering by adding a relation vector causally shifts generated descriptions (74.3% overall: Above/Below/Left 100%, Right 79%, In-front 62%, Behind 5%); validated on Llama-3.2-1B-Instruct and Qwen3-1.7B; geometry cleanest in PCA space at layer 24. [tehenan-etal-2025-spatial-world-models]

Structure

Linear Direction, Concept Crystals (Parallelogram/Trapezoid Structure)

Context

spatial reasoning, world models, orthogonal basis, antipodal negation, vector composition, relational structure, synthetic dataset, causal steering

Confirmed in models

Llama-3.2-3B-Instruct · Llama, Llama-3.2-1B-Instruct · Llama, Qwen3-1.7B · Qwen

Method

Linear probing, PCA, Activation Steering (Addition)

Papers

Linear Spatial World Models Emerge in Large Language Models — Tehenan, Matthieu, Moya, Christian Bolivar, Long, Tenghai, Lin, Guang2025 · arXiv:2506.02996