Nonlinear intrinsic dimension phase-transitions with emergent zero-shot competence

Lee et al. track per-layer nonlinear intrinsic dimension (TwoNN) and linear PCA effective dimension across pretraining of Pythia-410M/1.4B/6.9B [lee-etal-2024-geometric-signatures-of-compositionality-across-a-language-models-lifetime] On synthetic data both measures scale with dataset compositionality in fully-trained models, but nonlinear ID shows a sharp phase transition around step ~10^3 coinciding with the onset of zero-shot competence [lee-etal-2024-geometric-signatures-of-compositionality-across-a-language-models-lifetime] PCA effective dimension tracks superficial/Kolmogorov complexity (gzip compressibility, early-training Spearman rho near 1.0) while TwoNN ID never correlates with gzip [lee-etal-2024-geometric-signatures-of-compositionality-across-a-language-models-lifetime] The linear-superficial versus nonlinear-semantic dissociation generalizes to fully-trained Llama-3-8B and Mistral-7B [lee-etal-2024-geometric-signatures-of-compositionality-across-a-language-models-lifetime]