Definition

For a given attention head and query position $q$, treat the row of attention weights $a(q,\cdot)$ over relative key offsets $\delta = k - q$ as a discrete 1-D signal. Apply a Discrete Fourier Transform (or a wavelet decomposition for position-localized frequency content) to this signal, producing a power spectrum over frequency bands. Aggregating this per-head, per-layer spectrum across many query positions and inputs yields a frequency-band profile for each head: the fraction of its attention-weight energy concentrated in low frequencies (smooth, long-range, slowly-varying-with-offset attention) versus high frequencies (sharp, local, rapidly-varying-with-offset attention, e.g. attending strongly to only the immediately preceding token).

Relative to attention-graph-spectral-profile

Attention-graph spectral profile (Fiedler value, HFER, smoothness, spectral entropy) treats the attention matrix as the adjacency matrix of a graph over tokens and takes the spectrum of its graph Laplacian — a token-connectivity object, invariant to how attention weight varies smoothly or sharply as a function of relative position. This node instead treats a single head’s attention weights as a signal indexed by relative position and takes its Fourier/ wavelet spectrum — a position-signal object, orthogonal to the graph- connectivity question. Two attention patterns can have identical graph- Laplacian spectra while having very different relative-position frequency content (e.g. one uniformly-attending head and one head that alternates sharply between two fixed offsets), and vice versa.

Key evidence

Ruscio, Nanni & Silvestri (2025, ACL) apply DFT (Hann-windowed) and Daubechies-2 wavelet decomposition to per-head attention-weight rows, indexed by relative token position, across eight real pretrained LLMs (Gemma-2-2B, Pythia-2.8B/6.9B/12B, Llama-3.2-1B, Llama-3.1-8B, Mistral-7B, Qwen2.5-0.5B/5B) on 500 real Wikipedia sequences, finding individual heads reliably specialize as either local or global frequency processors — a “pronounced vertical striping” pattern where different heads consistently concentrate energy in distinct frequency bands (low frequencies carry 60-80% of spectral power on average; mid-band 15-25%; high-band 5-15%), stable across models and traced across Pythia-6.9B’s full training trajectory (step 0 to 143,000). A positional-encoding ablation (RoPE vs. T5 relative bias vs. BERT absolute PE vs. GPT-2 no PE) finds RoPE models show far tighter position-frequency coupling (scale-sensitivity 0.038) than T5 (0.627) or BERT (0.507), tying the effect specifically to RoPE’s mathematical structure while still measuring it empirically on real trained attention weights, not deriving it purely analytically. See ruscio-nanni-silvestri-2025-attention-heads-in-eight-real-pretrained-llms-reliably-specialize-into-consistent-frequency-bands-under-dft-and-wavelet-decomposition-of-their-position-indexed-attention-weights.

How to detect it

For each head, collect attention-weight rows $a(q,\cdot)$ across many query positions and inputs, reindex by relative offset $\delta=k-q$, and apply a DFT (or wavelet transform for position-localized analysis) to each row. Average the resulting power spectra across positions and inputs to get a per-head frequency profile; compare profiles across heads and layers to identify specialization, and check profile stability across different input distributions.

Key papers

• Ruscio, V., Nanni, U. & Silvestri, F. (2025). Beyond Position: the emergence of wavelet-like properties in Transformers. ACL 2025 / arXiv:2410.18067 — origin of DFT/wavelet-based per-head frequency-band specialization as a measurement distinct from graph-Laplacian attention spectra, and of the RoPE-specific position-frequency coupling finding.