Pretraining passes through three effective-rank phases
measured in 1 paper- Two spectral statistics of last-token representations - RankMe (von Neumann effective rank) and alpha-ReQ (eigenspectrum power-law exponent) - trace a consistent three-phase trajectory across OLMo and Pythia: warmup collapse, entropy-seeking expansion, then compression-seeking consolidation. [li-etal-2025-tracing-representation-geometry] - The entropy-seeking phase tracks peaking short-context n-gram memorization; the compression-seeking phase coincides with a grokking-like rise in downstream (TriviaQA/SciQ) accuracy. Across Pythia 1B-12B and OLMo2-1B, SciQ accuracy correlates with alpha-ReQ at r=0.54-0.84 (p<2.6e-3) and RankMe at r=-0.62 to -0.89. [li-etal-2025-tracing-representation-geometry] - Ablating all but the top-k eigendirections destroys downstream accuracy (Pythia-1B SciQ 0.838 to 0.225 at top-10), so the full spectrum is load-bearing; a proven cross-entropy + bottleneck (d<<|V|, Zipfian) derivation reproduces the phases, and removing either condition or using MSE eliminates compression-seeking. [li-etal-2025-tracing-representation-geometry] - Post-training diverges: SFT/DPO (Tulu-3 pipeline on Llama-3.1-8B) push entropy-seeking while RLVR pushes compression-seeking, with matching diversity-vs-accuracy tradeoffs; the pretraining phases are reported as correlational. [li-etal-2025-tracing-representation-geometry]