Sinusoidal number geometry recurs across eight LLMs and is causal

measured in 1 paper

Stefanik et al. show near-identical sinusoidal number geometry recurs across eight independently pretrained LLMs (OLMo 2, Llama 3, Phi 4 families) by both RSA and exact top-k Fourier-frequency overlap [stefanik-etal-2025-unravelling-number-manipulation-mechanisms] Number embeddings yield consistently higher RSA scores than random word-pieces, and the top k=63 Fourier frequencies show perfect agreement across all models [stefanik-etal-2025-unravelling-number-manipulation-mechanisms] The sinusoidal structure generalizes to natural-language numeric contexts, with probes over 70% accurate in all but three cases and over 90% for a majority [stefanik-etal-2025-unravelling-number-manipulation-mechanisms] Ablating the identified layers causally reduces arithmetic errors: division improves in all tested cases (27-64% error reduction) and multiplication in four of six cases [stefanik-etal-2025-unravelling-number-manipulation-mechanisms]

Structure

Lissajous Curves

Context

cross-model RSA and Fourier-frequency-IoU universality of sinusoidal number geometry, generalization of sinusoidal probing to natural-language numeric contexts, targeted layer ablation causally reduces multiplication/division error rates

Confirmed in models

OLMo 2 1B · OLMo 2, OLMo 2 7B · OLMo 2, OLMo 2 13B · OLMo 2, Llama-3.2-1B · Llama, Llama-3.2-3B · Llama, Llama-3-8B · Llama, Llama-3.1-8B · Llama, Phi-4 (15B) · Phi

Method

Structured-basis probing, Representational Similarity Analysis (RSA), Causal interventions (steering)

Papers

Unravelling the Mechanisms of Manipulating Numbers in Language Models — Štefánik, Michal, Mickus, Timothee, Kadlčík, Marek, Højer, Spiegel, Michal, Vázquez, Sinha, Kuchař, Josef, Mondorf, Philipp2025 · arXiv:2510.26285