Token-embedding intrinsic dimension is far below extrinsic dimension (about 13-25 for word2vec, GloVe and fastText; 25-122 for Pythia 14M-12B, redundancy 72-98 percent) and drops sharply early in Pythia training

- Kataiwa, Hakaze and Ohki apply a Levina-Bickel nearest-neighbour ID estimator to token embeddings: word2vec/GloVe/fastText (300d) have ID about 13-25 versus about 130 for a random Gaussian baseline, Pythia 14M-12B embedding layers have ID 25-122 against extrinsic dimension 128-5120 (redundancy 72-98 percent, about 98 percent from pythia-410m), and ID drops sharply in the first training steps of pythia-14m to 1.4b before stabilizing. LoRA rank near the ID gives a perplexity jump on pythia-410m. Word-vector checkpoints are mapped to the nearest existing entries (GloVe wiki-gigaword to glove-wikipedia, fastText wiki-news to fasttext-wiki-news-subwords-300). [kataiwa-etal-2025-measuring-intrinsic-dimension-of-token-embeddings]