Grokking Modular-Arithmetic Transformer

Research (Nanda, Chan, Lieberum, Smith & Steinhardt 2023; custom-trained from scratch on modular addition and related algorithmic tasks)

Structures found in this family (2)

Circle, Torus

By model (2)

Grokking Modular-Arithmetic Transformer (1 layer, 4 heads, d_model=128, mod 113)

Torus, Circle

Grokking Modular-Arithmetic Transformer (2 layers, 4 heads, pre-LN, d_model=128, mod 113/149/197)

Circle

Observations (3)

Fourier-circuit synchronization completes before grokking, causally advanceable

Modular-addition networks universally learn a torus projecting to a disc

Persistent-homology H1 cycle strength rises sharply at grokking

Papers

Circuit Synchronization Precedes Generalization: A Causal Precursor to Grokking (2026), Progress Measures for Grokking via Mechanistic Interpretability (2023), The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networks (2023), On the Geometry and Topology of Representations: The Manifolds of Modular Addition (2025), Topological Signatures of Grokking (2026)