Embedding intrinsic dimension expands early in pretraining, then compresses

Razzhigaev et al. apply TwoNN (cross-validated against Manifold-Adaptive Dimension Estimation and the Method of Moments) to embeddings sampled across real pretraining checkpoints of Bloom-3B and Pythia-2.8B [razzhigaev-etal-2024-shape-of-learning] Intrinsic dimension rises in the initial phase of training, then compresses toward the end, a training-time (not depth-wise) dimensionality trajectory [razzhigaev-etal-2024-shape-of-learning] The analysis is purely observational, with no causal intervention [razzhigaev-etal-2024-shape-of-learning]