MATH · IN · MODELS

Static embeddings carry the 'world-model' geographic signal

measured in 1 paper

Barenholtz re-tests Gurnee & Tegmark's linear space/time claim on static GloVe and Word2Vec embeddings using the same ridge-regression probes [barenholtz-2026-world-properties-without-world-models-recovering-spatial-and-temporal-structure-from-co-occurrence-statistics-in-static-word-embeddings] Geographic signal is strong (R^2=0.71-0.87 for latitude/longitude), temporal weaker (R^2=0.46-0.52), with elevation/GDP/population as unrecoverable controls [barenholtz-2026-world-properties-without-world-models-recovering-spatial-and-temporal-structure-from-co-occurrence-statistics-in-static-word-embeddings] Ablating PCA-derived country-name and climate-vocabulary subspaces causes large z-scored R^2 drops (up to z=25.9) far exceeding matched random-subspace controls [barenholtz-2026-world-properties-without-world-models-recovering-spatial-and-temporal-structure-from-co-occurrence-statistics-in-static-word-embeddings] Linear-probe recoverability alone therefore cannot distinguish genuine world-model structure from ordinary co-occurrence statistics [barenholtz-2026-world-properties-without-world-models-recovering-spatial-and-temporal-structure-from-co-occurrence-statistics-in-static-word-embeddings]

Context

a skeptical replication showing an "LLM world-model" finding also holds in non-contextual static embeddings with no sequential/attention mechanism at all, randomized-subspace-controlled PCA ablation distinguishing a genuine semantic-content causal effect from a generic capacity-loss artifact

Papers

World Properties without World Models: Recovering Spatial and Temporal Structure from Co-occurrence Statistics in Static Word Embeddings — Barenholtz, Elan2026 · arXiv:2603.04317