MATH · IN · MODELS
structures / Manifolds / 1D continuum manifold

1D continuum manifold

OBJECTgeometric-objectembedded-setK:zeroparametrizationmanifoldintermediatehow it's classified →

Open (non-periodic) one-dimensional manifold with extrinsic 'ripples' encoding an ordered continuous coordinate. Projected onto any two principal components it traces a Lissajous curve; higher-frequency harmonics are the ripples.

Replicationcomputed from the corpus — never hand-assigned
14 papers · no shared authors4 architecture classes · across papers4 domains · across papers14 model families · across papers
Filled = two or more values reported by papers that share no author — replication. Outlined = two or more values, but all from a single study — breadth, not replication. Grey = a single value. Derived from paper authorship and each model's architecture class, domain and family; it updates itself when a paper is added.

Definition

An open 1D manifold embedded in Rd\mathbb{R}^d: a smooth, non-self-intersecting curve (with endpoints, unlike the closed Circle) parametrised by a single ordered coordinate x∈[−1,1]x \in [-1,1]. A canonical form for such an embedding, when projected onto any two of its top principal components:

(w(μ)(xi), w(ν)(xi))=(aμsin⁡(kμxi), aνcos⁡(kνxi))\big(w^{(\mu)}(x_i),\, w^{(\nu)}(x_i)\big) = \big(a_\mu\sin(k_\mu x_i),\ a_\nu\cos(k_\nu x_i)\big)

for wavenumbers kμ<kνk_\mu < k_\nu. This is a Lissajous figure. The higher-frequency components (kν>kμk_\nu > k_\mu) are “ripples” — extrinsic curvature riding on top of the dominant, slowest-varying trend.

Intuition

A piece of string laid out along a wavy line through activation space, rather than a closed loop: it has two distinct endpoints, and — unlike a circle — walking far enough in one direction does not bring you back to the start.

Properties

  • H1=0H_1 = 0. An open arc is contractible (homotopy equivalent to a point), unlike a circle (H1(S1)=ZH_1(S^1)=\mathbb{Z}) — there is no non-contractible loop, since the curve has endpoints and never closes on itself.
  • Ordered coordinate. The parameter xx carries a genuine order (x1<x2x_1 < x_2 is meaningful), unlike the angular coordinate θ\theta on S1S^1, which is only defined modulo 2π2\pi and has no global “less than.”
  • Rational vs. irrational frequency ratio distinguishes closed from open. If kν/kμk_\nu/k_\mu is rational, the Lissajous projection is a closed curve that eventually repeats (the closed, periodic case belongs to Torus/Circle instead); if kν/kμk_\nu/k_\mu is irrational, the curve is quasi-periodic — it never exactly closes, densely covering a bounded region without repeating. This is what makes an open 1D manifold look like an endlessly “rippling,” never-quite-repeating curve.
  • Extrinsically curved, but the curvature is a projection artifact of higher harmonics, not an intrinsic property of the underlying 1-manifold (which, as any embedded arc, is intrinsically flat — an interval, isometric to a subset of R\mathbb{R} under its own induced arc-length metric, is what it is regardless of how it’s bent in the ambient space).
  • Amplitude decay with frequency. In the canonical form above, higher-wavenumber components typically carry smaller amplitude aνa_\nu — so the curve is dominated by its lowest-frequency component with progressively smaller “ripple” corrections from higher ones, rather than all frequencies contributing equally.

Key evidence

Burkhart, Ramadan, Liao, Chhikara, Rojas, Parker & Beaulieu-Jones (2025) train a real Llama-3.2-1B-architecture transformer from scratch on tokenized real EHR sequences (MIMIC-IV, CLIF-standardized), then transfer to a real independent hospital dataset (University of Chicago Medical Center). Two-component PCA of the model’s own decile/quantile token embeddings recovers the correct ordinal ranking of all ten deciles as a 1D continuum, alongside category-consistent clustering of clinical-concept tokens; separately, a linear-probe (logistic regression) analysis of patient-representation trajectory features over the first 24 hours of admission (path length, max single-step jump, an anomaly score) predicts four clinical outcomes (ROC-AUC 0.877-0.914 in MIMIC, degrading to 0.529-0.878 in the independent hospital before fine-tuning) — purely observational, no causal validation. See burkhart-etal-2025-ehr-fm-recovers-ordinal-1d-continuum-of-decile-embeddings-via-pca-and-trajectory-features-predict-clinical-outcomes.

Bassi & Tomar (2026) test whether Gurnee et al.’s (2025) curved-1D-manifold- plus-attention-head-twist motif for character counting in Claude 3.5 Haiku (see rippled-1d-continuum) generalizes across four new ordinal tasks and three open-weight models (Gemma-2-2B, Gemma-2-9B, Qwen3-4B). Tasks where the ordinal variable is locally computable from token identity (bracket nesting depth, markdown table column index) replicate a clean 1D manifold (>90% of per-value-centroid variance in PC1) with SAE features tiling it like place cells, pre-organized from the embedding layer (attention-head twist <=0.131 in every model); tasks requiring cross-position integration (indentation, table rows) need 2-4 principal components and are actively constructed by attention heads through training-dependent “twisting” (Qwen3-4B shows 4-10x higher twist than Gemma, 0.803 vs. 0.214-0.331 for indentation); and numeric magnitude — despite train R2>0.93R^2>0.93 — shows no coherent manifold at all (4-5 PCs needed, zero monotonic SAE features). Subspace ablation confirms causal necessity: patching the top-3 manifold directions drops probe accuracy/R2R^2 more than 28x more than patching random directions, across every task-model pair. See 1d-manifold-with-place-cell-tiling-generalizes-to-locally-computable-ordinal-tasks-across-architectures-but-not-to-numeric-magnitude-and-subspace-ablation-confirms-causal-necessity.

Singh (2026) proves via a minimum-norm/weight-decay argument (Proposition 1) that a transformer trained only on adjacent-pair comparisons over a total order must collapse embeddings onto a collinear, monotonically rank-ordered line with characteristic end-anchor stretching — a third distinct source of non-uniformity on this shape class, alongside the logarithmic compression of logarithmic-number-line and the oscillatory ripples of rippled-1d-continuum. Confirmed in a real, actually-trained single-layer toy Transformer (rrank=0.96r_{\rm rank}=0.96) and extended to real, pretrained Qwen2.5-1.5B across three ordinal domains (digits, sizes, months) with domain-dependent fidelity and near-orthogonal per-domain probe directions (|cos|<0.2); a cyclic control domain resists the line entirely. See singh-2026-a-minimum-norm-collinear-embedding-emerges-from-weight-decay-and-generalizes-to-real-qwen2-5-1-5b-ordinal-domains-with-near-orthogonal-per-domain-directions.

Exercises

Base

  1. Is an open arc {(t,0):t∈(0,1)}⊂R2\{(t, 0): t\in(0,1)\} \subset \mathbb{R}^2 homeomorphic to S1S^1? Justify using a topological invariant, not just intuition.
Solution

No. H1H_1 of the open arc is 00 (it’s contractible — homotopy equivalent to a point), while H1(S1)=Z≠0H_1(S^1) = \mathbb{Z} \neq 0. Since homology is a topological invariant, spaces with different H1H_1 cannot be homeomorphic.

  1. For the Lissajous parametrisation (aμsin⁡(kμx),aνcos⁡(kνx))(a_\mu\sin(k_\mu x), a_\nu\cos(k_\nu x)), what happens to the curve’s shape if kν=kμk_\nu = k_\mu (equal frequencies)?
Solution

With kν=kμ=kk_\nu=k_\mu=k: (aμsin⁡(kx),aνcos⁡(kx))(a_\mu\sin(kx), a_\nu\cos(kx)) traces an ellipse (a circle if aμ=aνa_\mu=a_\nu) as kxkx ranges over a full period — the simplest, 1:11{:}1-ratio Lissajous figure, which is a closed curve, not an open one. (This is the boundary case connecting to Circle/Torus; genuinely open, non-repeating behavior requires kν≠kμk_\nu \neq k_\mu with an irrational ratio.)

Middle

  1. Suppose kν/kμ=3/2k_\nu/k_\mu = 3/2 exactly (rational). Show that the curve (aμsin⁡(kμx),aνcos⁡(kνx))(a_\mu\sin(k_\mu x), a_\nu\cos(k_\nu x)) is periodic in xx, and find its period.
Solution

Write kμ=2ck_\mu = 2c, kν=3ck_\nu = 3c for some constant cc (matching the ratio 3/23/2). sin⁡(2c⋅x)\sin(2c\cdot x) has period π/c\pi/c in xx; cos⁡(3c⋅x)\cos(3c\cdot x) has period 2π/(3c)2\pi/(3c). The combined curve repeats when xx advances by a common multiple of both periods: lcm(π/c, 2π/(3c))\text{lcm}\big(\pi/c,\, 2\pi/(3c)\big). Writing both as multiples of π/(3c)\pi/(3c): π/c=3⋅π/(3c)\pi/c = 3\cdot\pi/(3c) and 2π/(3c)=2⋅π/(3c)2\pi/(3c) = 2\cdot \pi/(3c), so the lcm is 6⋅π/(3c)=2π/c6\cdot\pi/(3c) = 2\pi/c. The curve is periodic with period 2π/c2\pi/c — a closed (repeating), not open, curve, since the ratio is rational.

  1. Now suppose kν/kμk_\nu/k_\mu is irrational. Prove that the curve (aμsin⁡(kμx),aνcos⁡(kνx))(a_\mu\sin(k_\mu x), a_\nu\cos(k_\nu x)) never exactly returns to its starting point (aμsin⁡(kμx0),aνcos⁡(kνx0))(a_\mu\sin(k_\mu x_0), a_\nu\cos(k_\nu x_0)) for any x≠x0x \neq x_0.
Solution

Returning to the same point requires simultaneously kμx≡kμx0(mod2π)k_\mu x \equiv k_\mu x_0 \pmod{2\pi} and kνx≡kνx0(mod2π)k_\nu x \equiv k_\nu x_0 \pmod{2\pi}, i.e. kμ(x−x0)=2πmk_\mu(x-x_0) = 2\pi m and kν(x−x0)=2πnk_\nu(x-x_0) = 2\pi n for some integers m,nm,n (not both zero, since x≠x0x\neq x_0). Dividing: kν/kμ=n/mk_\nu/k_\mu = n/m, a ratio of integers — i.e. rational. This contradicts the assumption that kν/kμk_\nu/k_\mu is irrational (taking m≠0m\neq0; if m=0m=0 then x=x0x=x_0 already from the first equation, contradiction). Hence no such x≠x0x\neq x_0 exists — the curve never exactly repeats.

Pro

  1. Prove that if kν/kμk_\nu/k_\mu is irrational, the set of points {(kμx mod 2π, kνx mod 2π):x∈R}\{(k_\mu x \bmod 2\pi,\ k_\nu x \bmod 2\pi) : x \in \mathbb{R}\} is dense in the torus [0,2π)2[0,2\pi)^2 (this is the classical justification for “densely covers a bounded region without repeating”). You may use Weyl’s equidistribution theorem without proving it.
Solution

Rescale to u=kμx/2πu = k_\mu x/2\pi, so the point is (u mod 1, αu mod 1)(u \bmod 1,\ \alpha u \bmod 1) for α=kν/kμ\alpha = k_\nu/k_\mu irrational, tracing a line of irrational slope α\alpha on the unit torus [0,1)2[0,1)^2 as uu ranges over R\mathbb{R}. Weyl’s equidistribution theorem states that for irrational α\alpha, the sequence ({u},{αu})(\{u\}, \{\alpha u\}) (fractional parts) is equidistributed on [0,1)2[0,1)^2 as uu ranges over the integers (or, in the continuous analogue, the trajectory of the linear flow with irrational slope on the torus is equidistributed, hence dense). Density follows immediately from equidistribution: an equidistributed sequence/trajectory must enter every open subset of the torus (else that subset would have zero limiting frequency, contradicting equidistribution against the uniform measure, which assigns every nonempty open set positive measure). Rescaling back, the original curve is dense in [0,2π)2[0,2\pi)^2.

  1. A researcher fits the two-frequency model (aμsin⁡(kμx),aνcos⁡(kνx))(a_\mu\sin(k_\mu x), a_\nu\cos(k_\nu x)) to noisy empirical data and estimates k^ν/k^μ=1.6667\hat k_\nu/\hat k_\mu = 1.6667 from finite, noisy samples. Explain why this estimate can never distinguish “genuinely rational, e.g. 5/35/3” from “irrational but close to 5/35/3,” and what this implies about claiming open (non-periodic) structure from finite data.
Solution

Rational and irrational numbers are both dense in R\mathbb{R}, so for any irrational α\alpha there is a rational p/qp/q arbitrarily close to it (and vice versa) — no finite-precision measurement can distinguish a ratio that is exactly rational from one that is irrational but agrees with that rational to the measured precision, since doing so would require infinite precision (verifying agreement, or lack thereof, at every decimal place). Consequently, “open, non-repeating” structure is not something finite data can confirm in the strict topological sense (it would require infinite precision on the frequency ratio); in practice, the empirical claim has to be a weaker, operational one — e.g. “no exact repetition observed within the sampled range / at the achieved measurement precision,” or “the best-fit rational approximation has denominator large enough that the effective period exceeds the observed data range” — rather than a literal proof of irrationality.

Found in (10 observations · 12 families)

Gemma

Geometry of Ordinal Representations in Language Models (2026), When Models Manipulate Manifolds: The Geometry of a Counting Task (2025)measured

Locally-computable ordinals form SAE-tiled 1D manifolds; numeric magnitude does not

Details

Bassi & Tomar test whether the curved-1D-manifold-plus-attention-twist motif generalizes across four ordinal tasks and Gemma-2-2B/9B and Qwen3-4B [bassi-tomar-2026-geometry-of-ordinal-representations] Locally-computable variables (bracket-nesting depth, table column) concentrate over 90% of per-value-centroid variance in PC1, with Gemma Scope SAE features tiling the manifold like place cells [bassi-tomar-2026-geometry-of-ordinal-representations] Indentation needs 2-3 components and numeric magnitude needs 4-5 (PC1 only 40-62%) with zero monotonic SAE features, so magnitude is strongly linearly decodable (R^2>0.93) yet forms no coherent manifold [bassi-tomar-2026-geometry-of-ordinal-representations] Qwen3-4B shows 4-10x stronger attention-head twist than Gemma, but its indentation twisters preserve ordinal rank (Spearman >=0.77) while its numeric twisters do not (<=0.23) [bassi-tomar-2026-geometry-of-ordinal-representations] Activation patching of the top-3 manifold directions drops probe accuracy over 28x more than random directions, confirming the manifolds are informationally necessary, and the character-counting motif replicates the earlier Gurnee et al. finding [gurnee-etal-2025]

models: Gemma-2-2B, Gemma-2-9B · method: PCA, Activation patching, Sparse Autoencoders (SAE)
Latent Structure of Affective Representations in Large Language Models (2026)measured

Emotion representations trace a V-shaped open manifold with neutral at the vertex

Details

Choi & Weber build a classification-accuracy dissimilarity matrix from pairwise probes across GoEmotions categories at every layer of Gemma-2-9B and Mistral-7B, embedding it via MDS/Isomap/UMAP and Procrustes-aligning to human valence-arousal norms [choi-weber-2026-affective-structure] The recovered layout is a V-shaped parabolic open curve (an open 1-manifold, H1=0) with neutral at the vertex and positive/negative valence fanning out, rather than the classical circular circumplex [choi-weber-2026-affective-structure] Isomap's geodesic-to-Euclidean distance ratio (1.00-1.80 Gemma, 0.80-1.42 Mistral) confirms the curvature is real but modest [choi-weber-2026-affective-structure] Adding probe-direction vectors to Llama-3-70B-Instruct shifts human-rated output valence strongly (mean 7.29 vs 1.08 on a 10-point scale for positive versus negative targets) [choi-weber-2026-affective-structure]

models: Gemma-2-9B · method: Linear probing, Isomap, UMAP, Activation Steering (Addition)
Number Representations in LLMs: A Computational Parallel to Human Perception (2025), Shape Happens: Automatic Feature Manifold Discovery in LLMs via Supervised Multi-Dimensional Scaling (2025), Weber's Law in Transformer Magnitude Representations: Efficient Coding, Representational Geometry, and Psychophysical Laws in Language Models (2026), LLMs Know More About Numbers than They Can Say (2026)measured

Numbers lie on a logarithmically-compressed 1D number line

Details

Alquboj et al. project last-token numeric representations across six base models onto the top 1-2 PCA components, revealing a monotonic open number line whose spacing shrinks with magnitude (Scaling Rate Index beta<1), consistent with a logarithmic scale [alquboj-etal-2025] A target-aware PLS probe does not surface this compression while PCA's target-blind projection does, and the same compressed structure appears for birth years (not populations) in Llama-3 instruct models [alquboj-etal-2025] Tiblias et al. extend log-compression to temporal reasoning: durations and recurrence frequencies are best fit by a log-linear/log-semicircular manifold across Qwen2.5-3B, Llama-3.2-3B, and Gemma-2-2B, persisting to 70B scale [tiblias-etal-2025] Cacioli confirms log-compression across numerical, temporal, and spatial domains and three model families, with Weber R^2 up to .83-.88 versus linear .14-.32 [cacioli-2026-webers-law] A magnitude direction (a ridge probe, also the dominant PCA axis at r>.80 with log-magnitude) causally and dose-dependently shifts magnitude-comparison behavior at early layers (4.1x over random controls) [cacioli-2026-webers-law] The causal effect is layer-dissociated from geometry strength: at later layers where the structure is most pronounced the same intervention is inert (1.2x), and Mistral shows the geometry but not a human-range Weber fraction [cacioli-2026-webers-law] Yuchi et al. confirm log-magnitude compression via a supervised ridge probe across seven models, recovering numbers at ~2.3% relative error (rho=0.71 on real arXiv text) [yuchi-etal-2026-llms-know-more-about-numbers] They add a geometry-behavior dissociation: the hidden state linearly decodes a numeral pair's ranking at over 90% while verbalized comparison reaches only 50-70%, and an auxiliary-probe fine-tune closes this articulation gap by up to 84% [yuchi-etal-2026-llms-know-more-about-numbers]

models: Gemma-2-2B-it, Gemma-2-2B · method: PCA, Partial Least Squares (PLS), Geometric analysis, Supervised Multi-Dimensional Scaling (SMDS), Linear probing, Causal interventions (steering)
Symmetry in Language Statistics Shapes the Geometry of Model Representations (2026), When Models Manipulate Manifolds: The Geometry of a Counting Task (2025)measured

Ordered continuous concepts trace a rippled open 1D (Lissajous) manifold

Details

Karkada et al. show historical years lie on an open non-cyclic 1D manifold with higher-frequency ripple harmonics (a Lissajous curve in any two PCs), with amplitudes, phases, and frequencies analytically predicted and confirmed on Wikipedia embeddings and Gemma 2 2B for 1700-2020 [karkada-etal-2026] Gurnee et al. independently confirm the same curved helical open 1D shape with ringing cosine similarity for a character-counting feature in Claude 3.5 Haiku [gurnee-etal-2025] The two papers propose competing mechanisms, translation-symmetric co-occurrence statistics versus a capacity/distinguishability tradeoff, so they are candidate explanations for one geometry rather than confirmation of a single mechanism [karkada-etal-2026] Gurnee et al. additionally show attention heads causally twist the manifold to align a count with a target line-width, reducing the newline decision to a linear separator in a near-orthogonal subspace [gurnee-etal-2025]

models: Gemma-2-2B, EmbeddingGemma · method: Linear probing, PCA, Spectral analysis of the co-occurrence/PMI matrix, Analytical derivation, Co-occurrence statistics ablation, Sparse crosscoders, Circuit tracing (attribution graphs), Activation patching

Qwen

Geometry of Ordinal Representations in Language Models (2026), When Models Manipulate Manifolds: The Geometry of a Counting Task (2025)measured

Locally-computable ordinals form SAE-tiled 1D manifolds; numeric magnitude does not

Details

Bassi & Tomar test whether the curved-1D-manifold-plus-attention-twist motif generalizes across four ordinal tasks and Gemma-2-2B/9B and Qwen3-4B [bassi-tomar-2026-geometry-of-ordinal-representations] Locally-computable variables (bracket-nesting depth, table column) concentrate over 90% of per-value-centroid variance in PC1, with Gemma Scope SAE features tiling the manifold like place cells [bassi-tomar-2026-geometry-of-ordinal-representations] Indentation needs 2-3 components and numeric magnitude needs 4-5 (PC1 only 40-62%) with zero monotonic SAE features, so magnitude is strongly linearly decodable (R^2>0.93) yet forms no coherent manifold [bassi-tomar-2026-geometry-of-ordinal-representations] Qwen3-4B shows 4-10x stronger attention-head twist than Gemma, but its indentation twisters preserve ordinal rank (Spearman >=0.77) while its numeric twisters do not (<=0.23) [bassi-tomar-2026-geometry-of-ordinal-representations] Activation patching of the top-3 manifold directions drops probe accuracy over 28x more than random directions, confirming the manifolds are informationally necessary, and the character-counting motif replicates the earlier Gurnee et al. finding [gurnee-etal-2025]

models: Qwen3-4B · method: PCA, Activation patching, Sparse Autoencoders (SAE)
Number Representations in LLMs: A Computational Parallel to Human Perception (2025), Shape Happens: Automatic Feature Manifold Discovery in LLMs via Supervised Multi-Dimensional Scaling (2025), Weber's Law in Transformer Magnitude Representations: Efficient Coding, Representational Geometry, and Psychophysical Laws in Language Models (2026), LLMs Know More About Numbers than They Can Say (2026)measured

Numbers lie on a logarithmically-compressed 1D number line

Details

Alquboj et al. project last-token numeric representations across six base models onto the top 1-2 PCA components, revealing a monotonic open number line whose spacing shrinks with magnitude (Scaling Rate Index beta<1), consistent with a logarithmic scale [alquboj-etal-2025] A target-aware PLS probe does not surface this compression while PCA's target-blind projection does, and the same compressed structure appears for birth years (not populations) in Llama-3 instruct models [alquboj-etal-2025] Tiblias et al. extend log-compression to temporal reasoning: durations and recurrence frequencies are best fit by a log-linear/log-semicircular manifold across Qwen2.5-3B, Llama-3.2-3B, and Gemma-2-2B, persisting to 70B scale [tiblias-etal-2025] Cacioli confirms log-compression across numerical, temporal, and spatial domains and three model families, with Weber R^2 up to .83-.88 versus linear .14-.32 [cacioli-2026-webers-law] A magnitude direction (a ridge probe, also the dominant PCA axis at r>.80 with log-magnitude) causally and dose-dependently shifts magnitude-comparison behavior at early layers (4.1x over random controls) [cacioli-2026-webers-law] The causal effect is layer-dissociated from geometry strength: at later layers where the structure is most pronounced the same intervention is inert (1.2x), and Mistral shows the geometry but not a human-range Weber fraction [cacioli-2026-webers-law] Yuchi et al. confirm log-magnitude compression via a supervised ridge probe across seven models, recovering numbers at ~2.3% relative error (rho=0.71 on real arXiv text) [yuchi-etal-2026-llms-know-more-about-numbers] They add a geometry-behavior dissociation: the hidden state linearly decodes a numeral pair's ranking at over 90% while verbalized comparison reaches only 50-70%, and an auxiliary-probe fine-tune closes this articulation gap by up to 84% [yuchi-etal-2026-llms-know-more-about-numbers]

models: Qwen2.5-3B-Instruct, Qwen2.5-3B, Qwen2.5-7B-Instruct · method: PCA, Partial Least Squares (PLS), Geometric analysis, Supervised Multi-Dimensional Scaling (SMDS), Linear probing, Causal interventions (steering)
Emergent Ordinal Geometry in Transformers Trained on Local Comparisons (2026)measured

Weight decay collapses ordinal embeddings onto a monotone end-anchored line

Details

Singh proves (Proposition 1, a minimum-norm/weight-decay argument) that a transformer trained only on adjacent-pair comparisons over a total order collapses its embeddings onto a collinear, monotonically rank-ordered line with end-anchor stretching [singh-2026-emergent-ordinal-geometry] A trained single-layer toy transformer confirms this, its peak-checkpoint embeddings forming a monotone curve (rank correlation 0.96) with the predicted symbolic-distance and end-anchor effects [singh-2026-emergent-ordinal-geometry] Probing pretrained Qwen2.5-1.5B recovers the same monotone line across three ordinal domains with domain-dependent fidelity: digits cleanest (r~0.81 on PC2), sizes moderate, and months weakest (r=0.41) because a calendar is cyclic and resists the line [singh-2026-emergent-ordinal-geometry] Per-domain probe directions are near-orthogonal (|cos|<0.2), so distinct ordinal concepts occupy separate directions of the same shape [singh-2026-emergent-ordinal-geometry]

models: Qwen2.5-1.5B · method: PCA, Linear probing, Analytical derivation

Llama

Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behavior (2026)measured

Activation manifolds are isometric to behavior manifolds; geodesic steering wins

Details

Wurgaft et al. fit a second manifold over Llama-3.1-8B's output token distributions (in Hellinger/sqrt-probability space) alongside the residual-stream concept manifold [wurgaft-etal-2026-manifold-steering] Geodesic distances between the activation and behavior manifolds correlate at r=0.99 (weekdays), 0.89 (months), 0.999 (letters), and 0.999 (ages), far above the straight-line Euclidean distances (0.89/0.53/0.71/0.36) [wurgaft-etal-2026-manifold-steering] Manifold steering along the activation geodesic produces natural on-manifold behavioral trajectories where linear steering does not, and a pullback test recovers the activation manifold from a target behavior trajectory [wurgaft-etal-2026-manifold-steering] The approach generalizes to synthetic multi-dimensional concept spaces and a Mountain Car visual world-model task [wurgaft-etal-2026-manifold-steering]

models: Llama-3.1-8B · method: PCA, Geometric analysis, Activation Steering (Addition)
Latent Structure of Affective Representations in Large Language Models (2026)measured

Emotion representations trace a V-shaped open manifold with neutral at the vertex

Details

Choi & Weber build a classification-accuracy dissimilarity matrix from pairwise probes across GoEmotions categories at every layer of Gemma-2-9B and Mistral-7B, embedding it via MDS/Isomap/UMAP and Procrustes-aligning to human valence-arousal norms [choi-weber-2026-affective-structure] The recovered layout is a V-shaped parabolic open curve (an open 1-manifold, H1=0) with neutral at the vertex and positive/negative valence fanning out, rather than the classical circular circumplex [choi-weber-2026-affective-structure] Isomap's geodesic-to-Euclidean distance ratio (1.00-1.80 Gemma, 0.80-1.42 Mistral) confirms the curvature is real but modest [choi-weber-2026-affective-structure] Adding probe-direction vectors to Llama-3-70B-Instruct shifts human-rated output valence strongly (mean 7.29 vs 1.08 on a 10-point scale for positive versus negative targets) [choi-weber-2026-affective-structure]

models: Llama-3-70B-Instruct · method: Linear probing, Isomap, UMAP, Activation Steering (Addition)
Number Representations in LLMs: A Computational Parallel to Human Perception (2025), Shape Happens: Automatic Feature Manifold Discovery in LLMs via Supervised Multi-Dimensional Scaling (2025), Weber's Law in Transformer Magnitude Representations: Efficient Coding, Representational Geometry, and Psychophysical Laws in Language Models (2026), LLMs Know More About Numbers than They Can Say (2026)measured

Numbers lie on a logarithmically-compressed 1D number line

Details

Alquboj et al. project last-token numeric representations across six base models onto the top 1-2 PCA components, revealing a monotonic open number line whose spacing shrinks with magnitude (Scaling Rate Index beta<1), consistent with a logarithmic scale [alquboj-etal-2025] A target-aware PLS probe does not surface this compression while PCA's target-blind projection does, and the same compressed structure appears for birth years (not populations) in Llama-3 instruct models [alquboj-etal-2025] Tiblias et al. extend log-compression to temporal reasoning: durations and recurrence frequencies are best fit by a log-linear/log-semicircular manifold across Qwen2.5-3B, Llama-3.2-3B, and Gemma-2-2B, persisting to 70B scale [tiblias-etal-2025] Cacioli confirms log-compression across numerical, temporal, and spatial domains and three model families, with Weber R^2 up to .83-.88 versus linear .14-.32 [cacioli-2026-webers-law] A magnitude direction (a ridge probe, also the dominant PCA axis at r>.80 with log-magnitude) causally and dose-dependently shifts magnitude-comparison behavior at early layers (4.1x over random controls) [cacioli-2026-webers-law] The causal effect is layer-dissociated from geometry strength: at later layers where the structure is most pronounced the same intervention is inert (1.2x), and Mistral shows the geometry but not a human-range Weber fraction [cacioli-2026-webers-law] Yuchi et al. confirm log-magnitude compression via a supervised ridge probe across seven models, recovering numbers at ~2.3% relative error (rho=0.71 on real arXiv text) [yuchi-etal-2026-llms-know-more-about-numbers] They add a geometry-behavior dissociation: the hidden state linearly decodes a numeral pair's ranking at over 90% while verbalized comparison reaches only 50-70%, and an auxiliary-probe fine-tune closes this articulation gap by up to 84% [yuchi-etal-2026-llms-know-more-about-numbers]

models: Llama-2-7B, Llama-3.1-8B, Llama-3.2-1B, Llama-3.1-8B-Instruct, Llama-3.2-1B-Instruct, Llama-3.2-3B-Instruct, Llama-3.2-3B, Llama-3.1-70B-Instruct, Llama-3-8B-Instruct, Llama-3-8B · method: PCA, Partial Least Squares (PLS), Geometric analysis, Supervised Multi-Dimensional Scaling (SMDS), Linear probing, Causal interventions (steering)
Probing for Representation Manifolds in Superposition (2026)measured

A Manifold Probe recovers causally-used multi-dimensional time and space manifolds

Details

Modell introduces the Manifold Probe, which jointly learns (via a generalized eigenvalue problem over a spline basis) the space of a concept's linearly-decodable features and the orthonormal directions encoding them [modell-2026-manifold-probe] Applied to Llama-2-7B for release dates and geographic coordinates, it surfaces many more decodable features than the raw concept value, with the top SPACE feature more precisely decodable (higher test R^2) than latitude or longitude, while the top time feature is roughly identical to the year [modell-2026-manifold-probe] After Varimax rotation, top time features separate individual decades (1950s-2010s) and top space features localize on individual US states [modell-2026-manifold-probe] Treating the learned manifold as a continuum of steering vectors causally shifts the model's stated release year toward a target (peaking at layers 8 and 14), so the manifold is causally used, not merely decodable [modell-2026-manifold-probe]

models: Llama-2-7B · method: Linear probing, Activation Steering (Addition), Analytical derivation

Mistral

Latent Structure of Affective Representations in Large Language Models (2026)measured

Emotion representations trace a V-shaped open manifold with neutral at the vertex

Details

Choi & Weber build a classification-accuracy dissimilarity matrix from pairwise probes across GoEmotions categories at every layer of Gemma-2-9B and Mistral-7B, embedding it via MDS/Isomap/UMAP and Procrustes-aligning to human valence-arousal norms [choi-weber-2026-affective-structure] The recovered layout is a V-shaped parabolic open curve (an open 1-manifold, H1=0) with neutral at the vertex and positive/negative valence fanning out, rather than the classical circular circumplex [choi-weber-2026-affective-structure] Isomap's geodesic-to-Euclidean distance ratio (1.00-1.80 Gemma, 0.80-1.42 Mistral) confirms the curvature is real but modest [choi-weber-2026-affective-structure] Adding probe-direction vectors to Llama-3-70B-Instruct shifts human-rated output valence strongly (mean 7.29 vs 1.08 on a 10-point scale for positive versus negative targets) [choi-weber-2026-affective-structure]

models: Mistral-7B · method: Linear probing, Isomap, UMAP, Activation Steering (Addition)
Number Representations in LLMs: A Computational Parallel to Human Perception (2025), Shape Happens: Automatic Feature Manifold Discovery in LLMs via Supervised Multi-Dimensional Scaling (2025), Weber's Law in Transformer Magnitude Representations: Efficient Coding, Representational Geometry, and Psychophysical Laws in Language Models (2026), LLMs Know More About Numbers than They Can Say (2026)measured

Numbers lie on a logarithmically-compressed 1D number line

Details

Alquboj et al. project last-token numeric representations across six base models onto the top 1-2 PCA components, revealing a monotonic open number line whose spacing shrinks with magnitude (Scaling Rate Index beta<1), consistent with a logarithmic scale [alquboj-etal-2025] A target-aware PLS probe does not surface this compression while PCA's target-blind projection does, and the same compressed structure appears for birth years (not populations) in Llama-3 instruct models [alquboj-etal-2025] Tiblias et al. extend log-compression to temporal reasoning: durations and recurrence frequencies are best fit by a log-linear/log-semicircular manifold across Qwen2.5-3B, Llama-3.2-3B, and Gemma-2-2B, persisting to 70B scale [tiblias-etal-2025] Cacioli confirms log-compression across numerical, temporal, and spatial domains and three model families, with Weber R^2 up to .83-.88 versus linear .14-.32 [cacioli-2026-webers-law] A magnitude direction (a ridge probe, also the dominant PCA axis at r>.80 with log-magnitude) causally and dose-dependently shifts magnitude-comparison behavior at early layers (4.1x over random controls) [cacioli-2026-webers-law] The causal effect is layer-dissociated from geometry strength: at later layers where the structure is most pronounced the same intervention is inert (1.2x), and Mistral shows the geometry but not a human-range Weber fraction [cacioli-2026-webers-law] Yuchi et al. confirm log-magnitude compression via a supervised ridge probe across seven models, recovering numbers at ~2.3% relative error (rho=0.71 on real arXiv text) [yuchi-etal-2026-llms-know-more-about-numbers] They add a geometry-behavior dissociation: the hidden state linearly decodes a numeral pair's ranking at over 90% while verbalized comparison reaches only 50-70%, and an auxiliary-probe fine-tune closes this articulation gap by up to 84% [yuchi-etal-2026-llms-know-more-about-numbers]

models: Mistral-7B, Mistral-7B-Instruct-v0.3 · method: PCA, Partial Least Squares (PLS), Geometric analysis, Supervised Multi-Dimensional Scaling (SMDS), Linear probing, Causal interventions (steering)
The Origins of Representation Manifolds in Large Language Models (2025)measured

Cosine similarity approximates geodesic, not Euclidean, distance on feature manifolds

Details

Modell et al. model a feature as a metric space whose representation is a homeomorphic image on a manifold in the unit sphere, and prove that under a continuous-correspondence hypothesis cosine similarity approximates a monotone function of on-manifold geodesic distance rather than ambient Euclidean distance [modell-etal-2025] The theorem is validated on known manifold features via Kendall's tau and Chatterjee's xi between predicted and true geodesic distance [modell-etal-2025] The hue circle and day-of-year circle are found in OpenAI text-embedding-3-large raw embeddings, not SAE features [modell-etal-2025] The months circle is an SAE feature in Mistral-7B and the year axis is an SAE feature in GPT-2-small layer 7 (following Engels et al. 2025), isometric only on a log scale log(2019 - year) [modell-etal-2025]

models: Mistral-7B · method: Sparse Autoencoders (SAE), Geometric analysis

EHR Foundation Model (Llama-3.2-1B architecture, trained on MIMIC-IV)

Foundation Models for Electronic Health Records: Representation Dynamics and Transferability (2025)measured

A real EHR foundation model trained from scratch on MIMIC-IV recovers a correctly-ranked ordinal 1D continuum of decile embeddings via PCA

Details

Burkhart, Ramadan, Liao, Chhikara, Rojas, Parker & Beaulieu-Jones (2025) train a real Llama-3.2-1B-architecture transformer from scratch on tokenized real EHR sequences (MIMIC-IV, CLIF-standardized), then transfer to a real independent hospital dataset (UCMC); two-component PCA of the model's own decile/quantile token embeddings recovers the correct ordinal ranking of all ten deciles as a 1D continuum, alongside category-consistent clustering of clinical-concept tokens [burkhart-etal-2025-foundation-models-for-electronic-health-records] A linear-probe (logistic regression) analysis of patient-representation trajectory features over the first 24 hours of admission (path length, max jump, anomaly score) predicts four clinical outcomes (ROC-AUC 0.877-0.914 in MIMIC, degrading to 0.529-0.878 in the independent hospital before fine-tuning) -- purely observational, no causal validation [burkhart-etal-2025-foundation-models-for-electronic-health-records]

models: Custom Llama-3.2-1B-architecture transformer (trained from scratch on tokenized MIMIC-IV EHR sequences) · method:

Pythia

Number Representations in LLMs: A Computational Parallel to Human Perception (2025), Shape Happens: Automatic Feature Manifold Discovery in LLMs via Supervised Multi-Dimensional Scaling (2025), Weber's Law in Transformer Magnitude Representations: Efficient Coding, Representational Geometry, and Psychophysical Laws in Language Models (2026), LLMs Know More About Numbers than They Can Say (2026)measured

Numbers lie on a logarithmically-compressed 1D number line

Details

Alquboj et al. project last-token numeric representations across six base models onto the top 1-2 PCA components, revealing a monotonic open number line whose spacing shrinks with magnitude (Scaling Rate Index beta<1), consistent with a logarithmic scale [alquboj-etal-2025] A target-aware PLS probe does not surface this compression while PCA's target-blind projection does, and the same compressed structure appears for birth years (not populations) in Llama-3 instruct models [alquboj-etal-2025] Tiblias et al. extend log-compression to temporal reasoning: durations and recurrence frequencies are best fit by a log-linear/log-semicircular manifold across Qwen2.5-3B, Llama-3.2-3B, and Gemma-2-2B, persisting to 70B scale [tiblias-etal-2025] Cacioli confirms log-compression across numerical, temporal, and spatial domains and three model families, with Weber R^2 up to .83-.88 versus linear .14-.32 [cacioli-2026-webers-law] A magnitude direction (a ridge probe, also the dominant PCA axis at r>.80 with log-magnitude) causally and dose-dependently shifts magnitude-comparison behavior at early layers (4.1x over random controls) [cacioli-2026-webers-law] The causal effect is layer-dissociated from geometry strength: at later layers where the structure is most pronounced the same intervention is inert (1.2x), and Mistral shows the geometry but not a human-range Weber fraction [cacioli-2026-webers-law] Yuchi et al. confirm log-magnitude compression via a supervised ridge probe across seven models, recovering numbers at ~2.3% relative error (rho=0.71 on real arXiv text) [yuchi-etal-2026-llms-know-more-about-numbers] They add a geometry-behavior dissociation: the hidden state linearly decodes a numeral pair's ranking at over 90% while verbalized comparison reaches only 50-70%, and an auxiliary-probe fine-tune closes this articulation gap by up to 84% [yuchi-etal-2026-llms-know-more-about-numbers]

models: Pythia-2.8B · method: PCA, Partial Least Squares (PLS), Geometric analysis, Supervised Multi-Dimensional Scaling (SMDS), Linear probing, Causal interventions (steering)

GPT

Number Representations in LLMs: A Computational Parallel to Human Perception (2025), Shape Happens: Automatic Feature Manifold Discovery in LLMs via Supervised Multi-Dimensional Scaling (2025), Weber's Law in Transformer Magnitude Representations: Efficient Coding, Representational Geometry, and Psychophysical Laws in Language Models (2026), LLMs Know More About Numbers than They Can Say (2026)measured

Numbers lie on a logarithmically-compressed 1D number line

Details

Alquboj et al. project last-token numeric representations across six base models onto the top 1-2 PCA components, revealing a monotonic open number line whose spacing shrinks with magnitude (Scaling Rate Index beta<1), consistent with a logarithmic scale [alquboj-etal-2025] A target-aware PLS probe does not surface this compression while PCA's target-blind projection does, and the same compressed structure appears for birth years (not populations) in Llama-3 instruct models [alquboj-etal-2025] Tiblias et al. extend log-compression to temporal reasoning: durations and recurrence frequencies are best fit by a log-linear/log-semicircular manifold across Qwen2.5-3B, Llama-3.2-3B, and Gemma-2-2B, persisting to 70B scale [tiblias-etal-2025] Cacioli confirms log-compression across numerical, temporal, and spatial domains and three model families, with Weber R^2 up to .83-.88 versus linear .14-.32 [cacioli-2026-webers-law] A magnitude direction (a ridge probe, also the dominant PCA axis at r>.80 with log-magnitude) causally and dose-dependently shifts magnitude-comparison behavior at early layers (4.1x over random controls) [cacioli-2026-webers-law] The causal effect is layer-dissociated from geometry strength: at later layers where the structure is most pronounced the same intervention is inert (1.2x), and Mistral shows the geometry but not a human-range Weber fraction [cacioli-2026-webers-law] Yuchi et al. confirm log-magnitude compression via a supervised ridge probe across seven models, recovering numbers at ~2.3% relative error (rho=0.71 on real arXiv text) [yuchi-etal-2026-llms-know-more-about-numbers] They add a geometry-behavior dissociation: the hidden state linearly decodes a numeral pair's ranking at over 90% while verbalized comparison reaches only 50-70%, and an auxiliary-probe fine-tune closes this articulation gap by up to 84% [yuchi-etal-2026-llms-know-more-about-numbers]

models: GPT-2-Large · method: PCA, Partial Least Squares (PLS), Geometric analysis, Supervised Multi-Dimensional Scaling (SMDS), Linear probing, Causal interventions (steering)
The Origins of Representation Manifolds in Large Language Models (2025)measured

Cosine similarity approximates geodesic, not Euclidean, distance on feature manifolds

Details

Modell et al. model a feature as a metric space whose representation is a homeomorphic image on a manifold in the unit sphere, and prove that under a continuous-correspondence hypothesis cosine similarity approximates a monotone function of on-manifold geodesic distance rather than ambient Euclidean distance [modell-etal-2025] The theorem is validated on known manifold features via Kendall's tau and Chatterjee's xi between predicted and true geodesic distance [modell-etal-2025] The hue circle and day-of-year circle are found in OpenAI text-embedding-3-large raw embeddings, not SAE features [modell-etal-2025] The months circle is an SAE feature in Mistral-7B and the year axis is an SAE feature in GPT-2-small layer 7 (following Engels et al. 2025), isometric only on a log scale log(2019 - year) [modell-etal-2025]

models: GPT-2-small · method: Sparse Autoencoders (SAE), Geometric analysis

OpenAI Embeddings

The Origins of Representation Manifolds in Large Language Models (2025)measured

Cosine similarity approximates geodesic, not Euclidean, distance on feature manifolds

Details

Modell et al. model a feature as a metric space whose representation is a homeomorphic image on a manifold in the unit sphere, and prove that under a continuous-correspondence hypothesis cosine similarity approximates a monotone function of on-manifold geodesic distance rather than ambient Euclidean distance [modell-etal-2025] The theorem is validated on known manifold features via Kendall's tau and Chatterjee's xi between predicted and true geodesic distance [modell-etal-2025] The hue circle and day-of-year circle are found in OpenAI text-embedding-3-large raw embeddings, not SAE features [modell-etal-2025] The months circle is an SAE feature in Mistral-7B and the year axis is an SAE feature in GPT-2-small layer 7 (following Engels et al. 2025), isometric only on a log scale log(2019 - year) [modell-etal-2025]

models: text-embedding-3-large · method: Sparse Autoencoders (SAE), Geometric analysis

Custom Research CNN (purpose-built convolutional net for representation-geometry studies)

Number detectors spontaneously emerge in a deep neural network designed for visual object recognition (2019)measured

Numerosity-tuned units emerge, untrained, in a real CNN trained only for ImageNet object recognition, with Weber-Fechner logarithmic tuning geometry

Details

A real biologically-inspired deep CNN (8 convolutional layers, 5 max-pooling layers, 1 fully-connected layer) trained only on ILSVRC2012 ImageNet object classification develops units selectively tuned to abstract numerosity, despite never being trained on any numerosity task [nasr-viswanathan-nieder-2019-number-detectors-emerge-in-cnn] These numerosity-tuned units are concentrated in the network's final layers and drive the network's number-discrimination performance when read out [nasr-viswanathan-nieder-2019-number-detectors-emerge-in-cnn] The network's number discrimination shows the characteristic signature of the Weber-Fechner law -- discrimination accuracy depends on the ratio between numerosities rather than their absolute difference, matching human and animal number-discrimination psychophysics and implying a logarithmically-compressed tuning geometry along a discovered numerosity axis [nasr-viswanathan-nieder-2019-number-detectors-emerge-in-cnn]

models: 8-conv/5-maxpool/1-FC biologically-inspired CNN, trained on ILSVRC2012 ImageNet (Nasr, Viswanathan & Nieder) · method:

word2vec

Symmetry in Language Statistics Shapes the Geometry of Model Representations (2026), When Models Manipulate Manifolds: The Geometry of a Counting Task (2025)measured

Ordered continuous concepts trace a rippled open 1D (Lissajous) manifold

Details

Karkada et al. show historical years lie on an open non-cyclic 1D manifold with higher-frequency ripple harmonics (a Lissajous curve in any two PCs), with amplitudes, phases, and frequencies analytically predicted and confirmed on Wikipedia embeddings and Gemma 2 2B for 1700-2020 [karkada-etal-2026] Gurnee et al. independently confirm the same curved helical open 1D shape with ringing cosine similarity for a character-counting feature in Claude 3.5 Haiku [gurnee-etal-2025] The two papers propose competing mechanisms, translation-symmetric co-occurrence statistics versus a capacity/distinguishability tradeoff, so they are candidate explanations for one geometry rather than confirmation of a single mechanism [karkada-etal-2026] Gurnee et al. additionally show attention heads causally twist the manifold to align a count with a target line-width, reducing the newline decision to a linear separator in a near-orthogonal subspace [gurnee-etal-2025]

models: word2vec (trained on Wikipedia) · method: Linear probing, PCA, Spectral analysis of the co-occurrence/PMI matrix, Analytical derivation, Co-occurrence statistics ablation, Sparse crosscoders, Circuit tracing (attribution graphs), Activation patching

Claude

Symmetry in Language Statistics Shapes the Geometry of Model Representations (2026), When Models Manipulate Manifolds: The Geometry of a Counting Task (2025)measured

Ordered continuous concepts trace a rippled open 1D (Lissajous) manifold

Details

Karkada et al. show historical years lie on an open non-cyclic 1D manifold with higher-frequency ripple harmonics (a Lissajous curve in any two PCs), with amplitudes, phases, and frequencies analytically predicted and confirmed on Wikipedia embeddings and Gemma 2 2B for 1700-2020 [karkada-etal-2026] Gurnee et al. independently confirm the same curved helical open 1D shape with ringing cosine similarity for a character-counting feature in Claude 3.5 Haiku [gurnee-etal-2025] The two papers propose competing mechanisms, translation-symmetric co-occurrence statistics versus a capacity/distinguishability tradeoff, so they are candidate explanations for one geometry rather than confirmation of a single mechanism [karkada-etal-2026] Gurnee et al. additionally show attention heads causally twist the manifold to align a count with a target line-width, reducing the newline decision to a linear separator in a near-orthogonal subspace [gurnee-etal-2025]

models: Claude 3.5 Haiku · method: Linear probing, PCA, Spectral analysis of the co-occurrence/PMI matrix, Analytical derivation, Co-occurrence statistics ablation, Sparse crosscoders, Circuit tracing (attribution graphs), Activation patching

Ordinal Local-Comparison Toy Transformer

Emergent Ordinal Geometry in Transformers Trained on Local Comparisons (2026)measured

Weight decay collapses ordinal embeddings onto a monotone end-anchored line

Details

Singh proves (Proposition 1, a minimum-norm/weight-decay argument) that a transformer trained only on adjacent-pair comparisons over a total order collapses its embeddings onto a collinear, monotonically rank-ordered line with end-anchor stretching [singh-2026-emergent-ordinal-geometry] A trained single-layer toy transformer confirms this, its peak-checkpoint embeddings forming a monotone curve (rank correlation 0.96) with the predicted symbolic-distance and end-anchor effects [singh-2026-emergent-ordinal-geometry] Probing pretrained Qwen2.5-1.5B recovers the same monotone line across three ordinal domains with domain-dependent fidelity: digits cleanest (r~0.81 on PC2), sizes moderate, and months weakest (r=0.41) because a calendar is cyclic and resists the line [singh-2026-emergent-ordinal-geometry] Per-domain probe directions are near-orthogonal (|cos|<0.2), so distinct ordinal concepts occupy separate directions of the same shape [singh-2026-emergent-ordinal-geometry]

models: 1-Layer Ordinal Local-Comparison Transformer · method: PCA, Linear probing, Analytical derivation