Real pretrained transformers of varying scale share a nearly constant peak intrinsic dimension despite a wide range of ambient hidden sizes

- Mabrok (2026) models LLM hidden states as points on a latent semantic manifold equipped with the Fisher information metric, deriving a rate-distortion lower bound and a linear volume-scaling law for the "expressibility gap" (semantic distortion from discretizing continuous hidden states into a finite vocabulary) via the coarea formula. [mabrok-2026-latent-semantic-manifolds-llms] - Tested on six decoder-only checkpoints (GPT-2 124M, OPT-125M, Pythia-160M, GPT-2 XL, OPT-1.3B, Pythia-1B; Sec. 13.1) using ~1,800 WikiText-103 hidden-state vectors per layer, the peak TWO-NN intrinsic dimension is "remarkably consistent" at approximately 19-22 (Table 2: 19.2-22.1) despite ambient dimensions of 768-2048, i.e. a dimension-utilization ratio of 1-3% (Sec. 13.2). [mabrok-2026-latent-semantic-manifolds-llms] - The predicted linear scaling of the expressibility gap with the margin threshold epsilon (eta(epsilon) proportional to epsilon, not layer depth) is consistent with the six tested models' log-log fits (slopes 0.873-1.117, R^2>0.985; Sec. 13.4, Table 3), alongside an hourglass-shaped intrinsic-dimension depth profile in all six models (Sec. 13.2) and a 5th-percentile Voronoi margin that is nearly constant (0.04-0.06) across the six models (the paper's "hard core"; Sec. 13.4). The curvature estimates are small (order 1e-5, local-PCA proxy; Sec. 13.3). [mabrok-2026-latent-semantic-manifolds-llms]