Integers are represented as a multi-period helix used for addition

measured in 1 paper

Kantamneni & Tegmark find GPT-J-6B, Pythia-6.9B, and Llama-3.1-8B represent integer token values as a generalized helix: a linear axis plus cos/sin pairs at periods T=[2,5,10,100] [kantamneni-tegmark-2025-trigonometry-addition] The four periods are read directly from the layer-0 residual stream's own Fourier spectrum, tied to the base-10 numeral system [kantamneni-tegmark-2025-trigonometry-addition] Activation-patching reconstructions show the helix and circle fits recover most of addition's logit-difference effect (Helix 7.21, Circle 6.83 versus Full 8.34) and beat an equal-parameter non-periodic PCA baseline (6.13) [kantamneni-tegmark-2025-trigonometry-addition] The helical structure is load-bearing for addition and subtraction but not uniformly, underperforming a plain PCA baseline for division and multiplication [kantamneni-tegmark-2025-trigonometry-addition] They propose a Clock algorithm manipulating the a and b helices, extending Nanda et al.'s modular-addition Clock to general integer addition [kantamneni-tegmark-2025-trigonometry-addition]

Structure

Generalized Helix

Context

generalized helix representation of integers, combining a linear axis with cos/sin pairs at multiple periods (T=2,5,10,100), periods identified directly from the residual stream's own Fourier spectrum, tied to base-10 numeral structure, activation-patching comparison of Full/PCA/Helix/Circle/Polynomial reconstructions, quantifying how much of addition's causal effect each geometric form recovers, task-dependent load-bearing-ness of the helical structure (strong for addition/subtraction, weaker than a plain PCA baseline for division/multiplication), explicit extension of Nanda et al.'s "Clock" algorithm from one-layer modular-addition transformers to mid-sized LLMs performing general integer addition

Confirmed in models

GPT-J-6B · GPT-J, Pythia-6.9B · Pythia, Llama-3.1-8B · Llama

Method

Activation patching

Papers

Language Models Use Trigonometry to Do Addition — Kantamneni, Subhash, Tegmark, Max2025 · arXiv:2502.00873