Symmetric/skew-symmetric weight-matrix decomposition

methods / Theoretical / Analytical / Symmetric/skew-symmetric weight-matrix decomposition

Symmetric/skew-symmetric weight-matrix decomposition

Techniqueintermediate

Splits a learned square weight matrix (e.g. an attention query-key bilinear form) into its unique symmetric and skew-symmetric parts via the Toeplitz decomposition, then scores which part dominates by a Frobenius-norm ratio — a purely linear-algebraic diagnostic of whether a trained operator behaves as an order-independent or a genuinely directional interaction.

Used in (1 observation)

Encoder-only attention operators are measurably symmetric; decoder-only operators are measurably directionalmeasured

structure: Symmetric/skew decomposition of the attention query-key operator · models: Custom Transformer encoder (4-layer, trained from scratch), Custom Transformer encoder (12-layer, trained from scratch) · paper: The Underlying Structures of Self-Attention: Symmetry, Directionality, and Emergent Dynamics in Transformer Training