MATH · IN · MODELS

Saponati et al. (2025) Custom Encoder/Decoder Transformers

Research (Saponati, Sager, Vilimelis Aceituno, Stadelmann & Grewe; trained from scratch, arXiv:2502.10927)

Structures found in this family (1)

Symmetric/skew decomposition of the attention query-key operator

By model (2)

Custom Transformer encoder (12-layer, trained from scratch)
Symmetric/skew decomposition of the attention query-key operator
Custom Transformer encoder (4-layer, trained from scratch)
Symmetric/skew decomposition of the attention query-key operator

Observations (1)

Encoder-only attention operators are measurably symmetric; decoder-only operators are measurably directional

Papers

The Underlying Structures of Self-Attention: Symmetry, Directionality, and Emergent Dynamics in Transformer Training (2025)