Ordered continuous concepts trace a rippled open 1D (Lissajous) manifold

measured in 1 paper

Karkada et al. show historical years lie on an open non-cyclic 1D manifold with higher-frequency ripple harmonics (a Lissajous curve in any two PCs), with amplitudes, phases, and frequencies analytically predicted and confirmed on Wikipedia embeddings and Gemma 2 2B for 1700-2020 [karkada-etal-2026] Gurnee et al. independently confirm the same curved helical open 1D shape with ringing cosine similarity for a character-counting feature in Claude 3.5 Haiku [gurnee-etal-2025] The two papers propose competing mechanisms, translation-symmetric co-occurrence statistics versus a capacity/distinguishability tradeoff, so they are candidate explanations for one geometry rather than confirmation of a single mechanism [karkada-etal-2026] Gurnee et al. additionally show attention heads causally twist the manifold to align a count with a target line-width, reducing the newline decision to a linear separator in a near-orthogonal subspace [gurnee-etal-2025]

Structure

1D continuum manifold

Context

translation symmetry, historical time, ripples, Lissajous curves, linear coordinate decoding, character counting, ringing / Fourier modes, efficient / capacity-optimal coding, attention-head manifold twist

Confirmed in models

Gemma-2-2B · Gemma, EmbeddingGemma · Gemma, word2vec (trained on Wikipedia) · word2vec, Claude 3.5 Haiku · Claude

Method

Linear probing, PCA, Spectral analysis of the co-occurrence/PMI matrix, Analytical derivation, Co-occurrence statistics ablation, Sparse crosscoders, Circuit tracing (attribution graphs), Activation patching

Papers

Symmetry in Language Statistics Shapes the Geometry of Model Representations — Karkada, D., Korchinski, D. J., Nava, A., Wyart, M., Bahri, Y.2026 · arXiv:2602.15029

When Models Manipulate Manifolds: The Geometry of a Counting Task — Gurnee, W., Ameisen, E., Kauvar, I., Tarng, J., Pearce, A., Olah, C., Batson, J.2025 · arXiv:2601.04480