Attention heads in eight real pretrained LLMs reliably specialize into consistent frequency bands under DFT/wavelet decomposition

Ruscio, Nanni & Silvestri (2025, ACL) apply DFT and Daubechies-2 wavelet decomposition to per-head attention-weight rows, indexed by relative token position, across eight real pretrained LLMs (Gemma-2-2B, Pythia-2.8B/6.9B/12B, Llama-3.2-1B, Llama-3.1-8B, Mistral-7B, Qwen2.5-0.5B/5B) on 500 real Wikipedia sequences [ruscio-nanni-silvestri-2025-beyond-position-wavelet-like-properties-in-transformers] Individual heads reliably specialize as local or global frequency processors -- low frequencies carry 60-80% of spectral power on average, mid-band 15-25%, high-band 5-15% -- a pattern stable across models and traced across Pythia-6.9B's full training trajectory (step 0 to 143,000) [ruscio-nanni-silvestri-2025-beyond-position-wavelet-like-properties-in-transformers] A positional-encoding ablation (RoPE vs. T5 relative bias vs. BERT absolute PE vs. GPT-2 no PE) finds RoPE models show far tighter position-frequency coupling (scale-sensitivity 0.038) than T5 (0.627) or BERT (0.507), tying the effect specifically to RoPE's mathematical structure while still measuring it empirically on real trained attention weights [ruscio-nanni-silvestri-2025-beyond-position-wavelet-like-properties-in-transformers]