A sinusoidal probe decodes integers near-perfectly across nine LLMs

Kadlcik et al. test linear, log-linear, and fixed structured-basis (sinusoidal Fourier, binary-digit) probes for decoding integer values of number-token embeddings across nine LLMs (OLMo 2 1B/7B/13B/32B, Llama 3.2 1B/3B, Llama 3 8B/70B, Phi-4 15B) [kadlcik-etal-2025-number-embeddings] The fixed sinusoidal probe recovers integer values with near-perfect accuracy on most models and consistently outperforms the plain linear probe (<=0.05 everywhere), contradicting a purely-linear account of number encoding [kadlcik-etal-2025-number-embeddings] Sparse Fourier support and wave-like PCA projections confirm the periodicity is genuine, though OLMo 2 32B is a documented exception with near-zero probe accuracy and no periodic structure despite performing arithmetic accurately [kadlcik-etal-2025-number-embeddings] Tokens the probe fails to decode show higher downstream arithmetic error rates, and gradient-optimizing the four worst-decoded tokens in Llama 3 1B toward the sinusoidal pattern reduces their addition errors by 26% (17.6% to 13.0%) [kadlcik-etal-2025-number-embeddings]