Modular-addition networks (two-layer ReLU MLP, mod 97) compress onto a 2D cyclic (circle) geometry rather than the simplex ETF predicted by neural collapse

- While standard neural-collapse (NC) theory predicts that a K-class-balanced classifier organizes terminal representations as a (K-1)-dimensional simplex equiangular tight frame, Tan, Gai & Zhang (2026) report that a two-layer ReLU network with concatenated one-hot inputs trained on modular addition mod P = 97 instead compresses to an essentially two-dimensional subspace in which both token embeddings and last-layer classifier weights arrange as points on circles (Fig. 1B,C, PCA of embeddings and output weights; "the trained last-layer weight matrix has effective rank 2", Sec. 5.1). [tan-gai-zhang-2026-beyond-neural-collapse-modular-arithmetic] - Under stated assumptions (balanced classes, near-terminal feature-weight alignment, an additive feature model h_ij ~ A(e_i + e_j), prime modulus), they derive rather than measure a layerwise account: once the classifier has effective rank 2, all task gradients reaching the embeddings lie in the same plane (Prop. 5.4), the in-plane problem reduces to phase alignment on S^1 whose minimizers are single-frequency characters of Z/PZ, i.e. equal-angle points on a circle (Thm. 5.6). [tan-gai-zhang-2026-beyond-neural-collapse-modular-arithmetic] - They derive a trade-off between the cyclic rank-2 code and the simplex ETF: the cross-entropy gap is bounded (log I_0(1/tau) + o(1), O(1)), whereas the Schatten/nuclear-norm regularization saving is Theta(K), giving a threshold lambda_crit = Theta(1/K) (Thm. 6.6); for K = 97, q = 1, tau = 1 they compute lambda_crit ~ 2.86e-3 (Example 6.7). [tan-gai-zhang-2026-beyond-neural-collapse-modular-arithmetic]