Definition

For a single attention head with query and key projection matrices $W_Q, W_K \in \mathbb{R}^{d\times d_h}$, define the combined bilinear form $W_{qk} = W_Q W_K^\top \in \mathbb{R}^{d\times d}$, so that the pre-softmax attention logit between tokens $i$ and $j$ is $x_i^\top W_{qk} x_j$. Any square matrix admits a unique Toeplitz decomposition $W_{qk} = M_s + M_n$ with symmetric part $M_s = \tfrac12(W_{qk} + W_{qk}^\top)$ and skew-symmetric part $M_n = \tfrac12(W_{qk} - W_{qk}^\top)$. A symmetry score $s = 2(\|M_s\|_F^2 - \|M_n\|_F^2)/\|W_{qk}\|_F^2 \in [-1,1]$ quantifies which part dominates: $s>0$ means the operator is closer to a symmetric (undirected, content-similarity) bilinear form, $s<0$ means it is closer to a pure skew-symmetric (fully directional) form. A separate directionality score, built from the relative magnitude of outlier row- versus column-norms of $W_{qk}$, quantifies whether the operator’s dominant behavior is row- or column-driven — a distinct axis from the symmetric/skew split, capturing a different kind of asymmetry (which token’s representation dominates the interaction, rather than whether the interaction itself is order-independent).

Intuition

A symmetric bilinear form $x_i^\top M_s x_j = x_j^\top M_s x_i$ scores token pairs the same regardless of which token is “query” and which is “key” — attention driven purely by mutual content similarity, as in an undirected graph. A skew-symmetric form instead flips sign under swapping $i,j$, encoding a genuinely directional relationship (who attends to whom is not symmetric) — the natural fingerprint of positional order or causal precedence. Because $W_{qk}$ is learned freely rather than constrained to either extreme, measuring where a trained head actually sits on this spectrum reveals whether attention in that layer has organized itself around “what is similar” or “what comes before/after,” without assuming either in advance.

Properties

• Two independent axes, not one scalar. The symmetric/skew split answers “is the interaction order-independent,” while the row/column directionality score answers “which token’s representation dominates” — a head can score high on one axis and low on the other, so both must be reported, not collapsed into a single number.
• A structural signature that tracks architecture family, not just individual heads. Saponati, Sager, Vilimelis Aceituno, Stadelmann & Grewe (2025) compute the median symmetry and directionality score per layer across a broad sweep of real pretrained models — BERT (including BERT-Tiny), the GPT-2 family, LLaMA3, Phi, Mistral, ModernBERT, T5, plus vision and audio Transformers — finding encoder-only models consistently score higher on symmetry and decoder-only (causal) models consistently score higher on directionality, across every architecture and modality tested. See saponati-etal-2025-real-encoder-only-transformers-bert-modernbert-t5-encoder-score-measurably-higher-on-w-qk-symmetry-while-real-decoder-only-transformers-gpt-2-llama3-phi-mistral-score-higher-on-directionality.
• The asymmetry is provably gradient-driven, not incidental. The same paper proves (via a gradient decomposition of $W_{qk}$‘s update as a sum of rank-1 outer-product terms $x_i x_j^\top$) that causal masking during training systematically biases gradient updates toward the skew-symmetric part, giving a mechanistic account for why decoder-only training produces directional operators without any explicit symmetry constraint being imposed.
• Causally load-bearing, not merely descriptive. Initializing $W_Q, W_K$ so that $W_{qk}$ starts out symmetric (versus independent random initialization) measurably speeds convergence and lowers final loss in real encoder-only Transformers trained from scratch on real text corpora (Jigsaw, Wikipedia, Red Pajama) — up to 73% faster convergence for a 4-layer encoder on Wikipedia, with smaller but still positive effects at 12 layers — a direct intervention on the operator’s own symmetric/skew balance with a measured training- dynamics consequence, not just a post-hoc correlational characterization of already-trained weights.

Relative to continuous-causal-attention-operator

Continuous causal attention operator (CCT) generalizes the domain of attention (from discrete token positions to continuous time) while reusing a pretrained model’s weights unchanged. This node instead asks a structural question about the weights themselves — whether the already-existing, standard discrete attention operator’s $W_{qk}$ matrix has organized into a symmetric or directional bilinear form as a consequence of training. The two nodes concern the same family of objects (linear operators mediating the attention mechanism) but at different levels: one is about extending attention’s input domain, the other about classifying the algebraic structure of its learned weight matrix.

Key papers

• Saponati, M., Sager, P., Vilimelis Aceituno, P., Stadelmann, T. & Grewe, B. (2025). The Underlying Structures of Self-Attention: Symmetry, Directionality, and Emergent Dynamics in Transformer Training. arXiv:2502.10927 — origin of the symmetric/skew decomposition and directionality score, and the symmetric- initialization causal intervention.