The intrinsic dimension of linguistic datasets in OPT-350m, OPT-1.3b and OPT-6.7b predicts their information-theoretic coding length, and high compression predicts rapid adaptation

- Cheng, Kervadec and Baroni estimate the intrinsic dimension of last-token representations at every layer of pre-trained OPT-350m, OPT-1.3b and OPT-6.7b on many datasets, report dataset ID of order 10 (about two orders of magnitude below the ambient dimension 1024/2048/4096) with similar ranges across the three sizes, and find it positively Spearman-correlated with the coding length (perplexity) the LM assigns the data (rho 0.51 for OPT-6.7b with the ESS estimator, max over layers) and with ease of adaptation under fine-tuning (e.g. rho 0.72 with sample complexity). A battery of 12 ID estimators is compared; nearest-neighbour estimators track the relationship best, PCA and Fisher separability least (full text read; correlational, ID profiles over layers are mostly flat). [cheng-etal-2023-bridging-information-theoretic-and-geometric-compression-in-language-models]