OLMoE

Allen Institute for AI

Structures found in this family (2)

Linear Direction, Intrinsic-dimension profile across depth

By model (1)

OLMoE-1B-7B · 7B (1B active, MoE)

Linear Direction, Intrinsic-dimension profile across depth

Observations (2)

Concept directions anti-concentrate in the unembedding's low-variance spectral tail

A participation-ratio spectral signal finds attention circuits label-free across scale

Papers

Concepts Whisper While Syntax Shouts: Spectral Anti-Concentration and the Dual Geometry of Transformer Representations (2026), Spectral Probe-Circuits: A Three-Step Recipe for Identifying Attention-Head Circuits in Pretrained Transformers (2026)