The spectral edge of the update direction in grokking transformers collapses onto a low-dimensional subspace of functional modes, a single Fourier mode for modular addition

- Xu studies small from-scratch grokking transformers (2 layers, 4 heads, d_model=128, mod 97; plus shared-trunk multitask models). The leading directions of the weight-update trajectory (the "spectral edge") induce perturbation fields over the input domain that concentrate on one Fourier mode for modular addition (peak Fourier concentration 0.40 vs 0.021 uniform baseline) and, in a discrete-log basis, one mode for multiplication (5.9x concentration gain); subtraction spans several modes without separating from the bulk, and x^2+y^2 is only partly explained by additive-by-multiplicative cross terms (probe R^2 0.16). The "subspace" is a span of input-domain functions, not a subspace of activation space (the activation perturbation is high-rank, effective rank about 40 of 128); the paper limits itself to modular arithmetic in small transformers. [xu-2026-spectral-edge-dynamics-of-grokking]