Grokking Modular-Arithmetic MLP

Research (custom-trained from scratch on modular addition, MLP architecture as a non-attention grokking comparison point)

Structures found in this family (1)

Circle

By model (2)

Grokking Modular-Arithmetic MLP (3 hidden layers, width 512, d_embed=128, mod 113/149/197)

Circle

Two-layer ReLU MLP on modular addition (concatenated one-hot inputs, mod 97)

Circle

Observations (2)

Modular-addition networks (two-layer ReLU MLP, mod 97) compress onto a 2D cyclic (circle) geometry rather than the simplex ETF predicted by neural collapse

Persistent-homology H1 cycle strength rises sharply at grokking

Papers

Beyond Neural Collapse: Task-Intrinsic Geometry Governs Neural Representations in Modular Arithmetic (2026), Topological Signatures of Grokking (2026)