Across 28 open language models, the intrinsic dimension of hidden representations follows a layer-wise profile with a hunch in the middle layers during multiple-choice decision making

- Joshi, Bhatt and Modi estimate intrinsic dimension (MLE, TwoNN, GRIDE) of last-token hidden representations layer by layer across 28 open-weight transformer language models on multiple-choice tasks and report a consistent "hunchback" profile in MLP outputs: ID rises in early layers, peaks mid-network, then declines; the individual checkpoints are not tagged here because App. D / Table 4 list the 28 models (19 GPT-2/GPT-Neo/GPT-J/Phi/Gemma/Llama/Mistral variants plus 9 Pythia sizes) by name only, without exact checkpoint identifiers. [joshi-etal-2025-geometry-of-decision-making-in-language-models] - Residual-post activations show smoother, more monotonic ID changes than MLP outputs, and on MMLU-STEM the ID trend is flatter; the ID peak is reported to align with the onset of accurate logit-lens predictions, which the authors call correlational rather than causal. [joshi-etal-2025-geometry-of-decision-making-in-language-models] - In Pythia checkpoints on synthetic tasks the hunchback emerges during training, and more in-context examples lower final-layer ID. [joshi-etal-2025-geometry-of-decision-making-in-language-models]