Attention sinks form three position-encoding-linked reference-frame configurations

- Attention sinks act as geometric reference frames (distinguished token positions whose key vectors anchor every other token's query) and fall into three topologically distinct configurations that track the position-encoding scheme. [ruscio-nanni-silvestri-2025-attention-sink-geometry] - Centralized frames (standard RoPE: Llama, Mistral, Gemma) have a single dominant anchor (usually [BOS], ~30-40% attention), a star-like graph with stable Betti-0 (26.43, unchanged) and no Betti-1 cycles. [ruscio-nanni-silvestri-2025-attention-sink-geometry] - Distributed frames (NTK-scaled RoPE: Qwen2.5, Phi-2) have several weaker anchors (~10-15% each); bidirectional frames (absolute PE: BERT, XLM-RoBERTa) use two depth-shifting anchors, dropping Betti-0 22.68 to 1.69 and raising Betti-1 to 19.69. [ruscio-nanni-silvestri-2025-attention-sink-geometry] - Mean sink concentration is 82.93% (centralized), 69.92% (distributed) and 66.86% (bidirectional), up to 96.40% per layer; the architecture-frame association is significant in 92.5/90.0/82.5% of tests. [ruscio-nanni-silvestri-2025-attention-sink-geometry] - A Random Matrix Theory analysis of Pythia checkpoints (steps 0-143,000) shows the spectral-gap and participation-ratio signatures appear in the earliest training steps, before task loss converges, with a non-monotonic peak at 6.9B and a suggested phase transition between 6.9B and 12B. [ruscio-nanni-silvestri-2025-attention-sink-geometry] - Measured training-free (persistent homology / Betti numbers, Fiedler spectral connectivity, sink geometry) on 16 models across four position-encoding regimes; the paper cites Xiao et al. (2023) for sink-ablation effects rather than running its own causal test. [ruscio-nanni-silvestri-2025-attention-sink-geometry]