Decomposes weights or activations into a fixed harmonic (discrete Fourier) basis, rather than a data-fitted one, revealing sparse support on a handful of frequencies when the underlying computation is periodic — and enabling ablation directly in the frequency domain.
Used in (8 observations)
structure: Lissajous Curves · models: Modular-Arithmetic Vanilla RNN (1 layer, tanh, d_h=256, mod 113) · paper: Modular Addition in Recurrent Neural Networks Requires Low-Rank Fourier Circuits
structure: Circle · models: Small MLP/Transformer trained on finite-group composition tasks · paper: A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operations
structure: Circle · models: Grokking Modular-Arithmetic Transformer (2 layers, 4 heads, pre-LN, d_model=128, mod 113/149/197) · paper: Circuit Synchronization Precedes Generalization: A Causal Precursor to Grokking
structure: Lissajous Curves · models: GPT-2-XL · paper: Pre-trained Large Language Models Use Fourier Features to Compute Addition
structure: Circle · models: Small MLPs/Transformers trained on modular addition · paper: Uncovering a Universal Abstract Algorithm for Modular Addition in Neural Networks
structure: Torus, Circle · models: Grokking Modular-Arithmetic Transformer (1 layer, 4 heads, d_model=128, mod 113), Clock/Pizza Transformer, Model A (1 layer, constant attention alpha=0, width 128, mod 59), Clock/Pizza Transformer, Model B (1 layer, normal attention alpha=1, width 128, mod 59) · paper: Progress Measures for Grokking via Mechanistic Interpretability, The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networks, On the Geometry and Topology of Representations: The Manifolds of Modular Addition
structure: Circle · models: Swaroop ReLU MLP on modular arithmetic (1 hidden layer, width 256, mod 97) · paper: Latent Algorithmic Structure Precedes Grokking: A Mechanistic Study of ReLU MLPs on Modular Arithmetic
structure: Circle · models: 2-layer quadratic-activation MLP (widths 20/512/1024, modular addition mod p in {23,71,127}) · paper: Composing Global Solutions to Reasoning Tasks via Algebraic Objects in Neural Nets