Token point-cloud intrinsic dimension across layers correlates with next-token-prediction loss, and shuffling tokens raises the intrinsic-dimension peak

- Viswanathan et al. treat each layer's token hidden states as a point cloud and track its geometric properties -- intrinsic dimension, neighborhood overlap, cosine similarity -- across the depth of real Llama 3 8B, Mistral 7B, Pythia 6.9B (Deduplicated) and OPT 6.7B (each with 32 layers, 4096-dim hidden states) on 2244 Pile-10K prompts (first 1024 tokens each), mainly with the GRIDE estimator (ESS and TLE give qualitatively similar results). [viswanathan-etal-2025-geometry-of-tokens] - The token-cloud intrinsic dimension peaks in early-to-middle layers, and disrupting syntactic/semantic structure by shuffling the input tokens raises this intrinsic-dimension peak relative to unshuffled prompts, isolating a specific geometric signature of coherent versus incoherent input. [viswanathan-etal-2025-geometry-of-tokens] - Prompt-level cross-entropy loss on next-token prediction correlates with token geometry: prompts with higher loss have their tokens represented in higher-dimensional point clouds, directly linking the intrinsic-dimension profile of a real model's own token representations to its predictive uncertainty. [viswanathan-etal-2025-geometry-of-tokens]