Attention reference frame (sink-token anchor configuration)

structures / Linear Structures / Attention reference frame (sink-token anchor configuration)

Attention reference frame (sink-token anchor configuration)

OBJECTconfigurationadvancedhow it's classified →

A small, distinguished set of token positions R = {p_1,...,p_k} whose key vectors serve as fixed geometric anchors that every other token's query orients against, formalized as R = (M, P, phi): a coordinate chart on the representation manifold M. The attention-sink phenomenon (disproportionate attention to specific tokens) is the attention-weight signature of this configuration, not an architectural artifact.

Replicationcomputed from the corpus — never hand-assigned

1 paper2 architecture classes · one study1 domain8 model families · one study

Filled = two or more values reported by papers that share no author — replication. Outlined = two or more values, but all from a single study — breadth, not replication. Grey = a single value. Derived from paper authorship and each model's architecture class, domain and family; it updates itself when a paper is added.

Definition

A reference frame is a structure $\mathcal{R}=(\mathcal{M},\mathcal{P},\phi)$ where $\mathcal{M}$ is the representation manifold, $\mathcal{P}=\{p_1,\ldots,p_k\}$ is a small set of distinguished token positions (“reference points”), and $\phi:\mathcal{M}\times\mathcal{P}\to\mathbb{R}^d$ maps any point to its relationship with each reference point — a coordinate chart. An attention sink is the operational signature of a reference point: a token position $j$ for which $\mathrm{sink}(j) = \big[\tfrac1n\sum_{i=1}^n \mathbb{1}_{\{\alpha_{ij}\ge\tau\}}\big]\ge\gamma$ (a large fraction of source tokens $i$ assign it attention weight above a threshold $\tau$ , typically the 90th percentile). Unlike Linear Direction or Linear Subspace, which describe a single feature axis shared across activations, a reference frame is a configuration of points (an ordered, small tuple of key/query anchor tokens) that the rest of the sequence orients against via cosine-similarity/dot-product geometry — placing it in the same configuration role as Relation frame (ordered multi-token tuple geometry), though the two structures are unrelated (relation frame is about $k$ -argument semantic relation binding; this is about geometric coordinate anchoring).

Three configuration subtypes

Centralized: a single dominant reference point (typically [BOS]), a universal origin every token’s query aligns to with roughly constant attention weight (~30-40%) regardless of semantic content. Produces a star-like attention graph — a pointed manifold $(\mathcal{M}, p_0)$ .
Distributed: several weaker reference points (~10-15% attention each), a more flexible multi-anchor coordinate system.
Bidirectional: two anchors (sequence start and end) with layer-dependent weighting $\beta_j^{(l)}$ that shifts from start-dominant in early layers to end-dominant in deep layers — a dynamic coordinate system that changes through network depth.

Relative to linear-direction and linear-subspace

A reference frame is not itself a single fixed direction — it is the set of anchor points plus the mapping that orients other tokens relative to them. The eigendecomposition of the attention matrix $A\approx UV^\top$ shows dominant eigenvectors aligning with the reference tokens, so each subtype does induce an associated stable subspace (an associatedGeometry facet), but the primary object being classified is the configuration of anchor points itself, which is what varies (1 vs. several vs. 2 dynamically-reweighted) across architectures.

Key evidence

Ruscio, Nanni & Silvestri (2025) test this framework across eight model families spanning three position-encoding regimes: standard RoPE (Llama 3.1-8B/3.2-1B/3.2-3B, Mistral-7B-v0.1, Gemma-7B) develops centralized frames; NTK-aware scaled RoPE (Qwen2.5-3B/7B, Phi-2) develops distributed frames; absolute position embeddings (BERT-base, XLM-RoBERTa-large) develop bidirectional frames. Topological and spectral-graph statistics distinguish the three: Betti-0 connectivity is stable in centralized models (26.43 early vs. 26.43 late, unchanged) but drops sharply in bidirectional models (22.68 to 1.69) as the dynamic dual anchor reorganizes the attention graph; Betti-1 (cycle count) stays at 0 for centralized/distributed but rises to 19.69 (early) before falling to 3.11 (late) for bidirectional; the Fiedler value (algebraic connectivity) is far higher for bidirectional (38.3 to 22.5 across depth) than centralized (12.5 to 7.0) or distributed (16.2 to 11.0) graphs; and mean sink concentration is 82.93% (centralized), 69.92% (distributed), 66.86% (bidirectional), with per-layer maxima up to 96.40%. A Random Matrix Theory analysis of Pythia checkpoints (step 0 to step 143,000) finds the attention spectral-gap and participation-ratio changes are already present within the first several hundred training steps — reference frames emerge early in training as an optimal solution to the softmax probability-simplex constraint, well before task performance converges, non-monotonically across model scale (peaking at 6.9B, a possible phase transition between 6.9B and 12B). Correlational statistics (Pearson/ Spearman, t-tests) confirm these architecture-vs-frame-type associations are significant in the large majority of tested comparisons (92.5% centralized, 90.0% distributed, 82.5% bidirectional). The paper reports no causal ablation of its own (it cites Xiao et al. 2023’s prior finding that removing attention sinks degrades performance, but does not run that experiment itself). See transformer-attention-organizes-into-one-of-three-reference-frame-configurations-centralized-distributed-or-bidirectional-that-track-position-encoding-choice-and-emerge-within-the-first-training-steps.

Relative to attention-graph-spectral-profile

Attention-graph spectral profile (Fiedler value, HFER, smoothness, spectral entropy) uses the same attention-as-graph Laplacian toolkit (including the Fiedler value) but for a different purpose: a general four-diagnostic measurement correlated with mathematical-reasoning validity, rather than a configuration of distinguished anchor points.

How to detect it

Compute the per-token attention-weight distribution $\mathrm{sink}(j)$ across many prompts; positions exceeding the threshold in most prompts are candidate reference points. Classify the configuration by counting stable reference points and tracking their identity/weight across layers (constant single point = centralized; several moderate points = distributed; two points whose relative weight $\beta^{(l)}$ shifts with depth = bidirectional). Cross-check with topological statistics (Betti numbers, Fiedler value) computed on the token-by-token attention graph.

Key papers

Xiao, Tian, Chen, Han & Lewis (2023). Efficient Streaming Language Models with Attention Sinks. arXiv:2309.17453 — origin of the “attention sink” empirical observation and its performance relevance.
Ruscio, Nanni & Silvestri (2025). What are you sinking? A geometric approach on attention sink. arXiv:2508.02546 — the reference-frame unification and the three-subtype classification documented above.

Found in (1 observation · 8 families)

Llama

What are you sinking? A geometric approach on attention sink (2025)measured

Attention sinks form three position-encoding-linked reference-frame configurations

Details

- Attention sinks act as geometric reference frames (distinguished token positions whose key vectors anchor every other token's query) and fall into three topologically distinct configurations that track the position-encoding scheme. [ruscio-nanni-silvestri-2025-attention-sink-geometry] - Centralized frames (standard RoPE: Llama, Mistral, Gemma) have a single dominant anchor (usually [BOS], ~30-40% attention), a star-like graph with stable Betti-0 (26.43, unchanged) and no Betti-1 cycles. [ruscio-nanni-silvestri-2025-attention-sink-geometry] - Distributed frames (NTK-scaled RoPE: Qwen2.5, Phi-2) have several weaker anchors (~10-15% each); bidirectional frames (absolute PE: BERT, XLM-RoBERTa) use two depth-shifting anchors, dropping Betti-0 22.68 to 1.69 and raising Betti-1 to 19.69. [ruscio-nanni-silvestri-2025-attention-sink-geometry] - Mean sink concentration is 82.93% (centralized), 69.92% (distributed) and 66.86% (bidirectional), up to 96.40% per layer; the architecture-frame association is significant in 92.5/90.0/82.5% of tests. [ruscio-nanni-silvestri-2025-attention-sink-geometry] - A Random Matrix Theory analysis of Pythia checkpoints (steps 0-143,000) shows the spectral-gap and participation-ratio signatures appear in the earliest training steps, before task loss converges, with a non-monotonic peak at 6.9B and a suggested phase transition between 6.9B and 12B. [ruscio-nanni-silvestri-2025-attention-sink-geometry] - Measured training-free (persistent homology / Betti numbers, Fiedler spectral connectivity, sink geometry) on 16 models across four position-encoding regimes; the paper cites Xiao et al. (2023) for sink-ablation effects rather than running its own causal test. [ruscio-nanni-silvestri-2025-attention-sink-geometry]

models: Llama-3.1-8B, Llama-3.1-8B-Instruct, Llama-3.2-1B, Llama-3.2-3B · method: Geometric analysis