Transformer Jacobians grow non-normal with depth, collapsing effective rank

Fernando & Guitchounts compute exact per-sub-layer Jacobians of trained transformer blocks (Llama 3.1 8B, Gemma 4 E4B, OLMo 3 7B) in the residual stream, finding ~98% of eigenvalues occur as complex-conjugate pairs (rotational 2D subspaces invisible to SVD) [fernando-guitchounts-2026-transformer-jacobian-dynamics] A depth-wise non-normality gradient runs from near-pure-rotator early layers (self-alignment ~0.04) to near-symmetric late layers (~0.70) in Llama 3.1 8B, with Henrici departure falling 0.91 to 0.47 in lockstep [fernando-guitchounts-2026-transformer-jacobian-dynamics] The gradient is a joint architecture-and-training product: at OLMo's untrained checkpoint layers beyond the first are already near-symmetric (~0.95-0.98), and training pulls early/mid layers toward rotation [fernando-guitchounts-2026-transformer-jacobian-dynamics] Composing Jacobians across depth, effective rank collapses from ~436 (single layer) to 6.7 across Llama's 32 layers, leaving ~7 of 4,096 ambient directions [fernando-guitchounts-2026-transformer-jacobian-dynamics] A causal Schur surgery that holds eigenvalues fixed and scales only the non-normal part raises Llama's composed effective rank from 7.1 to 45.4 at dose zero (6.4x recovery), proving the bottleneck is caused by non-normality, not the eigenvalue spectrum [fernando-guitchounts-2026-transformer-jacobian-dynamics] The rank collapse is learned, not architectural: OLMo's untrained baseline stays nearly flat across all doses [fernando-guitchounts-2026-transformer-jacobian-dynamics] Signed-Leiden activation-correlation communities predict which units the Jacobian amplifies, with the coupling sign set by operator type (OLMo 0/32 to 24/32 significant layers over training; Gemma shows a sign inversion) [fernando-guitchounts-2026-transformer-jacobian-dynamics]