Token embedding spaces have non-constant local intrinsic dimension

- In the input token-embedding spaces of GPT-2, Llemma-7B, Mistral-7B and Pythia-6.9B, the local dimension around tokens, estimated from the slope of log-volume versus log-radius of the neighbour count, is highly variable (interquartile ranges in Table 1 span e.g. 20 to 531 for GPT-2 at small radius), and a statistical test rejects the hypothesis that the token subspace is a manifold at a small number of tokens in each model (Table 1: 68 of 50,257 tokens for GPT-2, 33 of 32,016 for Llemma-7B, 40 of 32,016 for Mistral-7B, 54 of 50,254 for Pythia-6.9B). [robinson-etal-2025-token-embeddings-violate-manifold-hypothesis] - The authors conclude that the token subspace is not globally a manifold, and for GPT-2, Llemma-7B and Mistral-7B also reject a smooth fiber-bundle null; they state that their test cannot say what kind of singularity is present. [robinson-etal-2025-token-embeddings-violate-manifold-hypothesis] - The rejections are identified only at specific tokens, and the pattern of rejected tokens differs between models even for Llemma-7B and Mistral-7B, which share an identical token set; the measurements are purely observational and concern input token embeddings, not hidden-layer representations. [robinson-etal-2025-token-embeddings-violate-manifold-hypothesis]