Grokking Modular-Arithmetic Transformer

Research (Nanda, Chan, Lieberum, Smith & Steinhardt 2023; custom-trained from scratch on modular addition and related algorithmic tasks)

Structures found in this family (3)

Circle, Torus, Linear Subspace

By model (5)

Grokking Modular-Arithmetic Transformer (1 layer, 4 heads, d_model=128, mod 113)

Torus, Circle

Grokking Modular-Arithmetic Transformer (2 layers, 4 heads, pre-LN, d_model=128, mod 113/149/197)

Circle

Grokking Multi-Task Modular-Arithmetic Transformer (shared-trunk, dual/tri-task) · ~303k-315k

Linear Subspace

Single-layer decoder-only Transformer on modular arithmetic (4 heads, d_model=128, d_mlp=512, mod 113)

Circle

Xu (2026) grokking Transformer (2 layers, 4 heads, d_model=128, d_ff=256, pre-norm, GELU, ~290k params, mod 97) · ~290k

Linear Subspace

Observations (5)

Fourier-circuit synchronization completes before grokking, causally advanceable

Single-layer transformers trained on modular arithmetic (mod 113) organize token embeddings into an approximately circular pattern in the top two principal components after generalization

Modular-addition networks universally learn a torus projecting to a disc

Persistent-homology H1 cycle strength rises sharply at grokking

The spectral edge of the update direction in grokking transformers collapses onto a low-dimensional subspace of functional modes, a single Fourier mode for modular addition

Papers

Circuit Synchronization Precedes Generalization: A Causal Precursor to Grokking (2026), Intrinsic Task Symmetry Drives Generalization in Algorithmic Tasks (2026), Progress Measures for Grokking via Mechanistic Interpretability (2023), The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networks (2023), On the Geometry and Topology of Representations: The Manifolds of Modular Addition (2025), Topological Signatures of Grokking (2026), Spectral Edge Dynamics Reveal Functional Modes of Learning (2026)