Definition

For a transformer block computing $h_{\ell+1}=h_\ell+f_\ell(h_\ell)$, the per-layer Jacobian is $J_\ell=\partial h_{\ell+1}/\partial h_\ell$, a linear operator on the residual stream. Any square matrix admits a complex Schur decomposition $J_\ell=Q_\ell(\Lambda_\ell+N_\ell)Q_\ell^*$ with $Q_\ell$ unitary, $\Lambda_\ell$ diagonal (the eigenvalues), and $N_\ell$ strictly upper-triangular — the operator’s non-normal part, zero exactly when $J_\ell$ is a normal matrix ($J_\ell J_\ell^*=J_\ell^*J_\ell$). Non-normality is quantified via the Henrici departure from normality, $\delta(J)=\sqrt{\|J\|_F^2-\sum_i|\lambda_i|^2}/\|J\|_F$, and via self-alignment, the overlap $\|V_{:,:k}^\top U_{:,:k}\|_F^2/k$ of the leading $k$ left/right singular subspaces — equal to $1$ for a symmetric (purely non-rotational) operator and $\approx k/d$ for a pure rotator. Distinct from both is the condition number $\kappa=\sigma_{\max}/\sigma_{\min}$, which measures anisotropic distortion but not rotational vs. symmetric operator type.

Relative to dimensional-collapse and intrinsic-dimension-profile

Dimensional collapse and Intrinsic-dimension profile across depth both track the effective rank of representation covariance (the cloud of activation vectors) — a property of the data distribution passing through the network. This node instead tracks the effective rank and non-normality of the Jacobian operators themselves (the linearized maps the network applies at each layer) — a property of the function the network computes, not of any particular batch of activations. The two can be connected (a collapsing composed-Jacobian rank constrains how much of the representation’s variance downstream layers can preserve) but are formally distinct objects.

Key evidence

Fernando & Guitchounts (2026) compute exact per-sub-layer Jacobians (not linearized approximations) on 1,000 WikiText-2 examples for Llama 3.1 8B (32 layers), Gemma 4 E4B (42 layers), and OLMo 3 7B at three training checkpoints (step 0/random init, step 471k, step 1.41M/final). Roughly 98% of eigenvalues occur as complex-conjugate pairs at every layer of every model — rotational 2D “spiral” subspaces invisible to SVD-only analysis. In Llama, self-alignment sweeps from $\approx0.04$ (near-pure rotator) in early layers to $\approx0.70$ in late layers while Henrici departure falls $0.91\to0.47$ — a genuine depth-wise gradient from rotation-dominated to near-symmetric operators — and this gradient is a joint architecture+training product: at OLMo step 0 (untrained, random weights, same wiring), layers beyond the first are already near-symmetric (self-alignment $\approx0.95$-$0.98$), and training pulls early/mid layers toward rotation (mid $0.95\to0.65$) while the late near-symmetric ceiling barely moves. Composing Jacobians across depth, $P_\ell=J_{31}\cdots J_\ell$, its effective rank $\mathrm{erank}(P)=\exp(-\sum_ip_i\log p_i)$ collapses from $\approx436$ (single layer) to $6.7$ (full 32-layer Llama composition, $\approx7$ surviving directions of 4,096); Gemma funnels to $5.9$ across 42 layers; OLMo’s composed effective rank drops two orders of magnitude over training ($\approx326$ at step 0 to $\approx42$ at step 1.41M).

Causal intervention (“Schur surgery”): holding $\Lambda_\ell,Q_\ell$ fixed and scaling only $N_\ell$ by a dose $c\in\{0,0.25,\ldots,2\}$ ($c=0$: fully normal at the trained eigenvalue spectrum; $c=1$: the trained model), then recomposing the full 32-layer stack and recomputing $\mathrm{erank}(P)$: Llama’s composed effective rank rises $7.1\to45.4$ (a 6.4x recovery) as $c\to0$, then falls again to $2.5$ at $c=2$ (over-amplified non-normality re-collapses rank); OLMo and Gemma reproduce the same monotone curve (6.6x and 5.7x recovery), while OLMo’s untrained step-0 baseline stays nearly flat across all doses — confirming the rank-bottleneck effect is learned, and that it is specifically the non-normal part $N_\ell$ of the Jacobian, not its eigenvalue spectrum $\Lambda_\ell$, that is causally responsible.

A second, related finding (Section 3): partitioning a sparse signed correlation graph over residual-stream units (top-20 edges/unit, signed Leiden community detection) into 6-87 mesoscale communities per layer, these communities’ coarse-grained projection of the Jacobian explains significantly more variance than size-matched random partitions at every layer of every model tested (even at OLMo step 0, median z=19.2 — a partly architectural prior), and a unit’s community-boundary (“bridging”) participation predicts how much the Jacobian amplifies it, with the sign of this coupling set by the layer’s operator type (self-alignment): OLMo goes from 0/32 FDR-significant layers (step 0) to 24/32 (step 1.41M, median Cohen’s d=+0.248); Llama shows 16/32 significant, uniformly positive; Gemma shows a sign inversion (4/42 positive, 13/42 negative, concentrated in strongly non-normal early/mid layers).

See trained-transformer-jacobians-develop-a-depth-wise-non-normality-gradient-and-a-cumulative-effective-rank-bottleneck-caused-by-the-non-normal-part-not-the-eigenvalue-spectrum-with-activation-community-structure-predicting-amplification-sign.

How to detect it

Compute exact per-layer Jacobians (autodiff, not finite differences) of a transformer block’s output w.r.t. its input; take the complex Schur decomposition; compute Henrici departure and leading-subspace self- alignment at each layer to characterize operator type, and the effective rank of the layer-composed product to check for a cumulative bottleneck. To test whether non-normality specifically (rather than the spectrum) is responsible for any observed rank collapse, hold the Schur eigenvalues fixed and dose-scale only the strictly-triangular part before recomposing — a “surgery” control unavailable to SVD-only or eigenvalue-only spectral analyses.

Key papers

• Fernando & Guitchounts (2026). Dynamics of the Transformer Residual Stream: Coupling Spectral Geometry to Network Topology. arXiv:2605.14258 — origin of the Jacobian non-normality gradient, the cumulative effective-rank bottleneck, and the Schur-surgery causal intervention documented above.