MATH · IN · MODELS

OLMo 2

Allen Institute for AI

Structures found in this family (5)

Linear Direction, Linear Subspace, Lissajous Curves, Linear Separability, Dimensional collapse

By model (8)

OLMo 2 7B · 7B
Lissajous Curves, Linear Direction, Linear Separability, Dimensional collapse, Linear Subspace
OLMo 2 1B · 1B
Lissajous Curves, Linear Direction, Dimensional collapse
OLMo 2 13B · 13B
Lissajous Curves, Linear Direction
OLMo 2 32B · 32B
Linear Direction, Lissajous Curves
OLMo 2 7B Instruct · 7B
Linear Subspace, Linear Direction
OLMo 2 13B Instruct · 13B
Linear Subspace
OLMo 2 1B Instruct · 1B
Linear Subspace
OLMo 2 32B Instruct · 32B
no structures recorded for this checkpoint specifically

Observations (10)

ICL answer tokens are affine functions in a PCA subspace
Boundless DAS finds causal subspaces for group-binding, recency and stereotype pronoun mechanisms; recency is distinct, binding and stereotype overlap
Sinusoidal number geometry recurs across eight LLMs and is causal
Cue-induced bias directions are installed by alignment tuning
Morphology stays linearly separable while lexical identity turns nonlinear with depth
Linear "modal difference vectors" that discriminate possible, improbable, impossible, and nonsensical sentences exist in language-model activations, emerge in a consistent order across training steps, layers, and model scale, and projections onto them track human categorization judgments
Pretraining passes through three effective-rank phases
A 40-dim SVD emotional subspace with four interpretable axes steers emotion
A sinusoidal probe decodes integers near-perfectly across nine LLMs
User gender, race and class directions from linear probes in four LLMs track population statistics and causally steer outputs

Papers

Functional Subspace, where language models can use vector algebra to solve problems (2026), A Mechanistic Understanding of Pronoun Fidelity in LLMs (2026), Unravelling the Mechanisms of Manipulating Numbers in Language Models (2025), How Does Alignment Tuning Shape Representations of Sycophancy and Related Cue-Induced Biases in LLMs? (2026), Finding Lexical Identity and Inflectional Morphology in Modern Language Models (2025), Is This Just Fantasy? Language Model Representations Reflect Human Judgments of Event Plausibility (2025), Tracing the Representation Geometry of Language Models from Pretraining to Post-training (2025), Emotions Where Art Thou: Characterizing the Emotional Latent Space of LLMs (2025), Pre-trained Language Models Learn Remarkably Accurate Representations of Numbers (2025), Linear socio-demographic representations emerge in Large Language Models from indirect cues (2025)