Clock/Pizza Modular-Arithmetic Transformer

Research (Zhong, Liu, Tegmark & Andreas 2023; custom-trained from scratch, large hyperparameter/initialization sweep on modular addition)

Structures found in this family (2)

Torus, Circle

By model (2)

Clock/Pizza Transformer, Model A (1 layer, constant attention alpha=0, width 128, mod 59)

Torus, Circle

Clock/Pizza Transformer, Model B (1 layer, normal attention alpha=1, width 128, mod 59)

Torus, Circle

Observations (1)

Modular-addition networks universally learn a torus projecting to a disc

Papers

Progress Measures for Grokking via Mechanistic Interpretability (2023), The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networks (2023), On the Geometry and Topology of Representations: The Manifolds of Modular Addition (2025)