Lissajous Curves

structures / Manifolds / Lissajous Curves

OBJECTgeometric-objectembedded-setparametrizationnon-manifoldintermediatehow it's classified →

A parametric curve traced by superposing sinusoids of different frequencies along different axes: (x(t), y(t)) = (A sin(at+δ₁), B sin(bt+δ₂)). The 2D shadow of any multi-frequency 1D manifold — the shared geometric signature behind manifolds-circle/torus and manifolds-1d-continuum.

Replicationcomputed from the corpus — never hand-assigned

6 papers · no shared authors3 architecture classes · across papers2 domains · across papers8 model families · across papers

Filled = two or more values reported by papers that share no author — replication. Outlined = two or more values, but all from a single study — breadth, not replication. Grey = a single value. Derived from paper authorship and each model's architecture class, domain and family; it updates itself when a paper is added.

Definition

A Lissajous curve is the parametric curve

$\big(x(t),\,y(t)\big) = \big(A\sin(at+\delta_1),\ B\sin(bt+\delta_2)\big)$

for frequencies $a,b > 0$ , amplitudes $A,B$ , and phases $\delta_1,\delta_2$ .

Intuition

Two pendulums swinging at slightly different rates, viewed from above, trace exactly this kind of curve — sometimes a simple closed loop, sometimes an intricate woven pattern, depending on the ratio of their frequencies.

Properties

Closed iff the frequency ratio is rational. If $b/a = p/q$ in lowest terms, the curve is periodic in $t$ with period $2\pi q/a$ (equivalently $2\pi p /b$ ) and closes into a curve with $p$ -fold or $q$ -fold symmetry (see 1D continuum manifold Exercise 3 for the derivation of the shared period from two individual periods).
Dense, non-closing if the frequency ratio is irrational. By Weyl equidistribution (see 1D continuum manifold Exercise 5), the trajectory never exactly repeats and is dense in the rectangle $[-A,A]\times[-B,B]$ .
$1{:}1$ ratio gives an ellipse (a circle if $A=B$ and $\delta_1-\delta_2=\pm\pi/2$ ). This is the simplest, fundamental-mode case — see Circle.
Every pair of coordinates of a translation-symmetric eigenmode expansion is a Lissajous curve. If an embedding takes the form $w(t) = (a_1\sin(k_1t), a_1\cos(k_1t), a_2\sin(k_2t), a_2\cos(k_2t),\ldots)$ — the general form of eigenmodes of a translation-symmetric (Toeplitz/circulant) kernel — then any pair of its coordinates is, by construction, a Lissajous curve with frequency ratio $k_i/k_j$ for whichever two modes are selected.
Symmetry under phase shift. Changing $\delta_1,\delta_2$ while keeping $a,b,A,B$ fixed rotates/reflects the curve’s pattern but preserves whether it is closed (rational ratio) or dense (irrational ratio) — phase does not affect the topological character, only the specific shape.

Key evidence

Zhou, Fu, Sharan & Jia (2024) show a fine-tuned GPT-2-XL solves addition using a sparse superposition of Fourier components (periods ~2, 2.5, 5, 10) in the residual stream — MLP layers concentrating low-frequency magnitude-approximating components, attention layers concentrating high-frequency modular-classification components — and confirm the causal role of each frequency band by low-pass/high-pass ablation, which produces the frequency-predicted error signature (off-by-10s for low-frequency removal, small errors for high-frequency removal). The same Fourier periods recur in the model’s own pre-trained token embeddings, and a model trained from scratch (no pre-training) shows no such structure and lower accuracy. See gpt-2-xl-mlp-attention-fourier-features-approximate-and-classify-addition.

Štefánik, Mickus, Kadlčík et al. (2025) extend Kadlčík et al.’s (2025) single-model sinusoidal-probe finding (above) into an explicit cross-model universality claim: seven independently pre-trained LLMs (OLMo 2 1B/7B/13B, Llama 3.2 1B/3B, Llama 3 8B, Phi-4 15B) show the same sinusoidal number geometry by two independent measures — Representational Similarity Analysis and exact overlap (IoU) of the top Fourier frequencies dominating each model’s number embeddings — and the same sinusoidal probe decodes numbers from natural-language contexts (arithmetic word problems, dates, medical codes, recipe quantities) at 70-90%+ accuracy. A causal layer-ablation intervention, removing the specific layers where numerical errors concentrate, reduces division errors by 27-64% and multiplication errors by up to 26%. See cross-model-rsa-fourier-iou-confirms-universal-sinusoidal-number-geometry-and-causal-layer-ablation-fixes-multiplication-division-errors.

Rangamani (2025) shows the same multi-frequency Fourier circuit found in transformers trained on modular addition (Nanda et al. 2023; Zhou et al. 2024, above) also emerges in a recurrent architecture: a single-layer tanh RNN (hidden size 256) trained from scratch on addition mod 113 develops low-rank weight matrices (rank 12-32 out of much higher ambient dimension, capturing 85-95% of each matrix’s energy) whose singular vectors pair up around 6 dominant frequencies. Sequential ablation of these frequencies — removing them one at a time — degrades accuracy monotonically from near-ceiling to chance once all six are removed, while any single frequency alone is dispensable. See a-1-layer-tanh-rnn-solving-modular-addition-mod-113-develops-rank-12-32-weight-matrices-whose-singular-vectors-pair-into-6-dominant-fourier-frequencies-and-sequential-frequency-ablation-collapses-accuracy-to-chance.

Exercises

Base

Write the Lissajous parametrisation for $a=b=1$ , $A=B=1$ , $\delta_1=0,\delta_2=\pi/2$ . What shape results?

Solution

$(x,y) = (\sin t, \sin(t+\pi/2)) = (\sin t, \cos t)$ — the unit circle, parametrised by $t$ .

For frequency ratio $b/a = 2/1$ (so $b=2a$ ), is the resulting curve closed? What is its period in $t$ ?

Solution

Yes, closed, since $2/1$ is rational. $x(t)=A\sin(at+\delta_1)$ has period $2\pi/a$ ; $y(t)=B\sin(2at+\delta_2)$ has period $2\pi/(2a)=\pi/a$ . The combined curve repeats at the least common multiple, $2\pi/a$ (since $2\pi/a$ is already a whole multiple — two periods — of $\pi/a$ ).

Middle

Show that a Lissajous curve with $A=B$ , $a=b$ , and phase difference $\delta_1-\delta_2 = 0$ (rather than $\pm\pi/2$ ) degenerates to a line segment rather than an ellipse.

Solution

With $a=b$ and $\delta_1=\delta_2=\delta$ : $x(t)=A\sin(at+\delta)$ , $y(t)=A\sin(at+\delta)$ — so $y(t)=x(t)$ identically for all $t$ . The “curve” is the line segment $\{(s,s): s\in[-A,A]\}$ , a degenerate (zero-area) Lissajous figure. This is the boundary case between the two families of ellipses traced out as $\delta_1-\delta_2$ varies from $0$ (degenerate line) to $\pi/2$ (circle/ellipse, maximal area) and back to $\pi$ (degenerate line again, opposite diagonal).

For frequency ratio $b/a = p/q$ in lowest terms (i.e. $\gcd(p,q)=1$ ), the closed curve is often described as having “ $p$ -fold” symmetry along one axis. Explain, by counting zero-crossings, why $x(t) = A\sin(at+\delta_1)$ completes $q$ full oscillations while $y(t)=B\sin(bt+\delta_2)$ completes $p$ full oscillations over one full period of the combined curve.

Solution

From the period analysis (as in 1D continuum manifold Exercise 3, generalized): the combined period is $T = 2\pi q/a = 2\pi p /b$ (using $b/a=p/q \Rightarrow b = ap/q$ , so $2\pi p/b = 2\pi p/(ap/q) = 2\pi q /a$ , confirming both expressions for $T$ agree). Over this period, $x(t)$ (with angular frequency $a$ ) completes $\frac{a\cdot T}{2\pi} = \frac{a\cdot 2\pi q/a}{2\pi} = q$ full cycles, and $y(t)$ (angular frequency $b$ ) completes $\frac{b\cdot T}{2\pi} = \frac{b\cdot 2\pi q/a}{2\pi} = \frac{bq}{a} = \frac{(ap/q)q}{a}=p$ full cycles. So $x$ oscillates $q$ times and $y$ oscillates $p$ times per shared period — the source of the “ $p$ -by- $q$ ” lobe/symmetry count in the classic Lissajous figures.

Pro

Prove that if $w(t) = (a_1\sin(k_1t), a_1\cos(k_1 t), a_2\sin(k_2t), a_2\cos(k_2t))$ (a 4D embedding, two “modes”) and $k_2/k_1$ is irrational, then the pair of coordinates $(a_1\sin(k_1t),\, a_2\sin(k_2t))$ is dense in $[-a_1,a_1]\times[-a_2,a_2]$ , but the full 4D trajectory does not densely fill a 4-dimensional region — explain what lower-dimensional set it’s confined to and why.

Solution

The 2D projection’s density follows directly from Weyl equidistribution applied to $(k_1t \bmod 2\pi,\ k_2t\bmod 2\pi)$ (as in 1D continuum manifold Exercise 5). But the full 4D trajectory is confined to the product of two circles of radii $a_1,a_2$ — i.e. a genuine 2-dimensional torus $T^2$ embedded in $\mathbb{R}^4$ (each pair $(a_1\sin k_1t, a_1\cos k_1t)$ always satisfies $x_1^2+x_2^2=a_1^2$ , and likewise for the second pair) — not a dense subset of the full 4-dimensional ambient space. Density (from irrational frequency ratio) and dimension-of-the-confining-set (from the sinusoidal functional form itself) are two separate facts: irrationality makes the trajectory dense within the 2-torus it’s confined to, but does not let it escape that 2-dimensional confining surface. This is the same distinction as in Torus Exercise 4: the shape of the occupied set ( $T^2$ , from the functional form) is a different question from whether a given trajectory fills that shape (density, from the frequency ratio).

Two Lissajous curves have the same frequency ratio $b/a = 2/3$ but different amplitude ratios $B/A$ . Are the two curves similar (related by a uniform scaling), congruent (related by an isometry), or neither in general? Justify with an example.

Solution

Neither, in general. Example: curve 1 has $A=B=1$ ; curve 2 has $A=1, B=2$ . Both trace closed curves with the same $3$ -by- $2$ lobe topology (from Exercise 4, since the ratio $2/3$ is shared), but curve 2 is stretched by a factor of $2$ along the $y$ -axis relative to curve 1 — a non-uniform (anisotropic) scaling. A uniform scaling (similarity) would require scaling both axes by the same factor, which does not turn curve 1 into curve 2 unless $B/A$ also matches. So matching frequency ratio guarantees the same topological lobe pattern, but not the same shape up to similarity or congruence — amplitude ratio is an independent parameter controlling the aspect ratio of the bounding rectangle.

Found in (4 observations · 5 families)

Rangamani (2025) Modular-Addition RNN

Modular Addition in Recurrent Neural Networks Requires Low-Rank Fourier Circuits (2025)measured

A 1-layer RNN solves modular addition via six Fourier frequencies

Details

Rangamani trains a single-layer tanh RNN from scratch on modular addition mod 113 to grokking, then SVDs each weight matrix and finds a few singular directions capture nearly all energy (rank 12 for W_E/W_fc/W_ih, rank 32 for W_hh) [rangamani-2025-fourier-rnn-modular-addition] Just 6 dominant Fourier frequencies (k in {6,15,20,29,47,54}) account for nearly all signal energy, and the 12 significant singular vectors of the unembedding group into exactly 6 frequency-dominated pairs [rangamani-2025-fourier-rnn-modular-addition] This reproduces the multi-frequency Fourier circuit previously found only in transformers inside a recurrent architecture instead [rangamani-2025-fourier-rnn-modular-addition] Removing any single frequency barely hurts, but sequential ablation degrades accuracy monotonically (28.3% after 4, 6.2% after 5, chance 0.885% after all 6), and restricting weights to the orthogonal complement of the low-rank subspace also drops accuracy to chance [rangamani-2025-fourier-rnn-modular-addition]

models: Modular-Arithmetic Vanilla RNN (1 layer, tanh, d_h=256, mod 113) · method: Fourier analysis of weights and activations, Causal interventions (steering)

OLMo 2

Unravelling the Mechanisms of Manipulating Numbers in Language Models (2025)measured

Sinusoidal number geometry recurs across eight LLMs and is causal

Details

Stefanik et al. show near-identical sinusoidal number geometry recurs across eight independently pretrained LLMs (OLMo 2, Llama 3, Phi 4 families) by both RSA and exact top-k Fourier-frequency overlap [stefanik-etal-2025-unravelling-number-manipulation-mechanisms] Number embeddings yield consistently higher RSA scores than random word-pieces, and the top k=63 Fourier frequencies show perfect agreement across all models [stefanik-etal-2025-unravelling-number-manipulation-mechanisms] The sinusoidal structure generalizes to natural-language numeric contexts, with probes over 70% accurate in all but three cases and over 90% for a majority [stefanik-etal-2025-unravelling-number-manipulation-mechanisms] Ablating the identified layers causally reduces arithmetic errors: division improves in all tested cases (27-64% error reduction) and multiplication in four of six cases [stefanik-etal-2025-unravelling-number-manipulation-mechanisms]

models: OLMo 2 1B, OLMo 2 7B, OLMo 2 13B · method: Structured-basis probing, Representational Similarity Analysis (RSA), Causal interventions (steering)

Pre-trained Language Models Learn Remarkably Accurate Representations of Numbers (2025)measured

A sinusoidal probe decodes integers near-perfectly across nine LLMs

Details

Kadlcik et al. test linear, log-linear, and fixed structured-basis (sinusoidal Fourier, binary-digit) probes for decoding integer values of number-token embeddings across nine LLMs (OLMo 2 1B/7B/13B/32B, Llama 3.2 1B/3B, Llama 3 8B/70B, Phi-4 15B) [kadlcik-etal-2025-number-embeddings] The fixed sinusoidal probe recovers integer values with near-perfect accuracy on most models and consistently outperforms the plain linear probe (<=0.05 everywhere), contradicting a purely-linear account of number encoding [kadlcik-etal-2025-number-embeddings] Sparse Fourier support and wave-like PCA projections confirm the periodicity is genuine, though OLMo 2 32B is a documented exception with near-zero probe accuracy and no periodic structure despite performing arithmetic accurately [kadlcik-etal-2025-number-embeddings] Tokens the probe fails to decode show higher downstream arithmetic error rates, and gradient-optimizing the four worst-decoded tokens in Llama 3 1B toward the sinusoidal pattern reduces their addition errors by 26% (17.6% to 13.0%) [kadlcik-etal-2025-number-embeddings]

models: OLMo 2 1B, OLMo 2 7B, OLMo 2 13B, OLMo 2 32B · method: Structured-basis probing

Llama

Unravelling the Mechanisms of Manipulating Numbers in Language Models (2025)measured

Sinusoidal number geometry recurs across eight LLMs and is causal

Details

models: Llama-3.2-1B, Llama-3.2-3B, Llama-3-8B, Llama-3.1-8B · method: Structured-basis probing, Representational Similarity Analysis (RSA), Causal interventions (steering)

Pre-trained Language Models Learn Remarkably Accurate Representations of Numbers (2025)measured

A sinusoidal probe decodes integers near-perfectly across nine LLMs

Details

models: Llama-3.2-1B, Llama-3.2-3B, Llama-3-8B, Llama-3-70B · method: Structured-basis probing

Phi

Unravelling the Mechanisms of Manipulating Numbers in Language Models (2025)measured

Sinusoidal number geometry recurs across eight LLMs and is causal

Details

models: Phi-4 (15B) · method: Structured-basis probing, Representational Similarity Analysis (RSA), Causal interventions (steering)

Pre-trained Language Models Learn Remarkably Accurate Representations of Numbers (2025)measured

A sinusoidal probe decodes integers near-perfectly across nine LLMs

Details

models: Phi-4 (15B) · method: Structured-basis probing

GPT

Pre-trained Large Language Models Use Fourier Features to Compute Addition (2024)measured

Fine-tuned GPT-2-XL solves addition with sparse Fourier features

Details

Zhou et al. fine-tune GPT-2-XL on addition (99.74% test accuracy) and use the logit lens to show it progressively refines its answer layer by layer rather than retrieving a memorized fact [zhou-etal-2024-fourier-features-for-addition] Fourier-transforming MLP and attention logits reveals sparse outlier components with periods ~2, 2.5, 5, 10, the same periods present in the pretrained token embeddings [zhou-etal-2024-fourier-features-for-addition] MLP layers concentrate low-frequency magnitude-approximating components while attention layers concentrate high-frequency modular-classification components [zhou-etal-2024-fourier-features-for-addition] Fourier-domain ablation confirms causality: removing high-frequency attention/MLP components drops accuracy to 27.08% with small-magnitude errors, and removing low-frequency MLP components drops it to 35.89% with off-by-10/50/100 errors [zhou-etal-2024-fourier-features-for-addition] A from-scratch model shows no Fourier structure and lower accuracy (94.44%), while injecting pretrained embeddings rescues both the structure and the accuracy [zhou-etal-2024-fourier-features-for-addition]

models: GPT-2-XL · method: Fourier analysis of weights and activations, Causal interventions (steering), Logit Lens