Training drives real Pythia layers to self-organize into stable low-/high-complexity curvature-entropy modules, with the low-curvature module best predicting brain alignment

- Liu, Ma, Tang, Gan, Zhang, Li, Xie & Song compute per-sentence von Neumann entropy of the token Gram matrix and curvature (mean turning angle of the token-to-token hidden-state trajectory) for every layer of Pythia 70M, 160M, 410M and 1B at 19 log-spaced training checkpoints (steps 1 to 143K) on 1,000 sentences; for Pythia-1B, k-means (K=2, chosen a priori) on each layer's entropy/curvature trajectory gives a stable partition into a low-complexity module (layers 4-15) and a high-complexity module (other layers), stable under leave-one-checkpoint-out and bootstrap resampling. [liu-etal-2026-training-driven-modularization] - The low-complexity module has lower entropy and curvature than the high-complexity module (mean Cohen's d = 2.42 and 2.64), with the gap emerging after about step 4K; the modular partition is shown in detail for Pythia-1B, and the paper reports it as a clustering result, not as evidence against a smooth depth gradient. [liu-etal-2026-training-driven-modularization] - Ridge-regression encoding scores for the TUCKUTE2024 fMRI language-network data (5 participants, 1,000 sentences) are higher for the low-complexity module in all ROIs; lower curvature remains a negative predictor of encoding score after controlling for training progress and layer effects (clearly detectable only at 1B: 6/6 ROIs negative, 4/6 FDR-significant), a correlational finding that the authors say does not establish causality. [liu-etal-2026-training-driven-modularization]