Single-layer transformers trained on modular arithmetic (mod 113) organize token embeddings into an approximately circular pattern in the top two principal components after generalization

- Hwang & Park train single-layer decoder-only Transformers (4 heads, d_model=128, d_mlp=512) from scratch on modular arithmetic with p=113 (training fraction 0.3, weight decay 1) and report that "after models achieve generalization, their embeddings consistently display a structured, low-dimensional geometric organization" (Sec. 4; configuration in App. F). [hwang-park-2026-intrinsic-task-symmetry-drives-generalization-in-algorithmic-tasks] - For modular arithmetic "the leading two principal components form an approximately circular pattern" in a PCA of the token embeddings (Fig. 1), which they describe as consistent with prior observations in modular addition; the evidence is a qualitative PCA visualisation, not a fitted or quantified circle. [hwang-park-2026-intrinsic-task-symmetry-drives-generalization-in-algorithmic-tasks] - Higher PC subspaces are described as "a helical organization, where each PC pair forms a circular projection with a systematic phase shift" (Sec. 4); the paper's Proposition 7.1 derives (proof in App. H, not measured) that embeddings obeying commutativity and associativity lie on a closed helix, i.e. S^1 embedded in a high-dimensional torus. [hwang-park-2026-intrinsic-task-symmetry-drives-generalization-in-algorithmic-tasks] - They propose a three-stage dynamic (memorization, symmetry acquisition, geometric organization) in which generalization emerges alongside symmetry acquisition and the circular/helical geometry becomes apparent only after the effective embedding dimensionality has decreased; the models are tiny from-scratch networks, not pretrained LLMs. [hwang-park-2026-intrinsic-task-symmetry-drives-generalization-in-algorithmic-tasks]