Layer-wise intrinsic dimension of token representations expands to a working space then contracts onto roughly 10-dimensional submanifolds across Qwen2.5 and Mistral-7B, and working-space dimension correlates negatively with parameter-sensitive performance

- Song et al. track a token-level "correlator" (normalized mean pairwise dot product of token vectors in a sequence) layer by layer in Qwen2.5, DeepSeek-R1-Distill-Qwen, Llama and Mistral models and, from an assumed projection model (the correlator times (dimension - 1) is approximately constant, calibrated on a randomly initialised copy of each model), convert it into dimension estimates. The correlator first falls and then rises across depth, read as an expansion into a high-dimensional "working space" (d_model of order 10^2) followed by contraction onto a lower-dimensional semantic submanifold (d_machine of order 10^1, "approximately 10-dimensional" in the abstract), with d_machine growing roughly linearly with parameter count. For models of the same size and architecture, lower d_model goes with better benchmark performance (a small number of models, read from Figure 6); the authors call the dimension identifications conjectures resting on simplifying assumptions, and the dimension is not measured with a standard ID estimator such as TwoNN. Only Qwen2.5-3B, Qwen2.5-7B and Mistral-7B are tagged (Llama and DeepSeek-R1-Distill-Qwen sizes and base/instruct variants are only in figure labels, and the paper does not state the Mistral or Qwen variants). [song-etal-2025-bridging-the-dimensional-chasm-uncover-layer-wise-dimensional-reduction-in-transformers]