Algorithmic phase diagnostics (gradient symmetricity, distance irrelevance)

methods / Theoretical / Analytical / Geometric analysis / Algorithmic phase diagnostics (gradient symmetricity, distance irrelevance)

Algorithmic phase diagnostics (gradient symmetricity, distance irrelevance)

Techniqueadvanced

Two automatable scalar metrics — gradient symmetricity and distance irrelevance — that classify which of several candidate algorithms a network implements, given that its embeddings already share the same base geometry (e.g. a circle) but combine it differently.

Used in (3 observations)

Fourier-circuit synchronization completes before grokking, causally advanceablemeasured

structure: Circle · models: Grokking Modular-Arithmetic Transformer (2 layers, 4 heads, pre-LN, d_model=128, mod 113/149/197) · paper: Circuit Synchronization Precedes Generalization: A Causal Precursor to Grokking

Modular-addition networks universally learn a torus projecting to a discmeasured

structure: Torus, Circle · models: Grokking Modular-Arithmetic Transformer (1 layer, 4 heads, d_model=128, mod 113), Clock/Pizza Transformer, Model A (1 layer, constant attention alpha=0, width 128, mod 59), Clock/Pizza Transformer, Model B (1 layer, normal attention alpha=1, width 128, mod 59) · paper: Progress Measures for Grokking via Mechanistic Interpretability, The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networks, On the Geometry and Topology of Representations: The Manifolds of Modular Addition

Modular-arithmetic MLPs form square-wave weights that alone generalizemeasured

structure: Circle · models: Swaroop ReLU MLP on modular arithmetic (1 hidden layer, width 256, mod 97) · paper: Latent Algorithmic Structure Precedes Grokking: A Mechanistic Study of ReLU MLPs on Modular Arithmetic