Encoder-only attention operators are measurably symmetric; decoder-only operators are measurably directional

Saponati, Sager, Vilimelis Aceituno, Stadelmann & Grewe decompose the query-key bilinear form W_qk of real pretrained Transformer attention heads into symmetric and skew-symmetric parts and measure a per-layer symmetry score across BERT, BERT-Tiny, ModernBERT, T5-encoder, GPT-2, LLaMA3, Phi, Mistral, and vision/audio Transformers, finding encoder-only models consistently score higher on symmetry and decoder-only models consistently score higher on a companion row/column directionality score [saponati-etal-2025-underlying-structures-self-attention-symmetry-directionality] A gradient-decomposition proof shows causal masking systematically biases W_qk's training updates toward the skew-symmetric part, giving a mechanistic account of why decoder-only training produces directional operators without any explicit symmetry constraint [saponati-etal-2025-underlying-structures-self-attention-symmetry-directionality] Initializing W_Q, W_K so W_qk starts symmetric, versus independent random initialization, causally speeds convergence and lowers final loss in real 4-layer and 12-layer Transformer encoders trained from scratch on Jigsaw, Wikipedia, and Red Pajama, with the largest effect a 73% faster convergence for a 4-layer encoder on Wikipedia [saponati-etal-2025-underlying-structures-self-attention-symmetry-directionality]