Definition
Rather than describing one manifold’s fixed shape, this concept describes how the manifold’s own intrinsic dimension (ID) — the minimum number of coordinates needed to describe the data locally without loss, measured here via the TwoNN nearest-neighbor estimator — changes as a function of network depth. Across the self-supervised transformers studied so far, the same qualitative trajectory recurs: an early expansion phase (ID rises sharply, the neighbor structure churns rapidly layer-to-layer), a compression phase (ID falls and stabilizes at a much lower plateau or local minimum, with high layer-to-layer neighbor stability), and — in some models — a final re-expansion/decoding phase as the representation is unpacked back toward the reconstruction target.
Relative to a single fixed manifold
Every other node under Manifolds (circle, torus, sphere, paraboloid, …) describes one manifold’s shape at a fixed site. This concept instead tracks a sequence of manifolds, one per layer, and treats their changing dimensionality itself as the object of interest — an encoder/decoder-style trajectory through representation space, rather than a static geometric claim about any one layer.
Key evidence
Valeriani, Doimo, Cuturello, Laio, Ansuini & Cazzaniga (2023) measure ID (Intrinsic dimension estimation (TwoNN)) and layer-to-layer neighbor stability (Neighborhood overlap) across ESM-2 protein language models (35M/650M/3B) and iGPT image transformers (S/M/L, 76M/455M/1.4B), finding the same expansion-then-compression shape in every model, with the ID’s local minimum in each case coinciding with the layer where neighborhood overlap with ground-truth semantic labels (remote protein homology; ImageNet class) peaks — i.e. the geometrically most-compressed layer is also the most semantically abstract one, in every model tested. A brief preliminary appendix extends this to Llama-2-70B on sentiment classification (SST): the ID profile is more complex (three peaks, two local minima) than in either protein or image models, but the qualitative link holds — overlap with the sentiment label partition is highest at the first local ID minimum. See Relation frame (ordered multi-token tuple geometry) and Separability/alignment decomposition for other depth-wise geometric decompositions in this corpus, on different structural objects (relation tuples; separability/alignment) than raw manifold dimensionality.
Extension to generative-latent and molecular-embedding manifolds
Choi, Hwang, Cho & Kang (2023) extend this concept from a per-layer
scalar profile to a spatially-varying field over a single real trained
GAN’s (StyleGAN/StyleGAN2) latent manifold: local intrinsic dimension is
estimated at each latent point via a Riemannian-manifold treatment of
the generator, and correlates strongly with supervised disentanglement
scores despite requiring no attribute labels. See
choi-etal-2023-local-intrinsic-dimension-of-a-real-stylegan2-latent-manifold-varies-across-latent-space-and-correlates-with-disentanglement.
El-Samman, Husain, Huynh, De Castro, Morton & De Baerdemacker (2024)
measure a related global-effective-dimension finding on a real trained
SchNet-family GNN’s molecular embeddings (QM9): the fully-trained
128-parameter embedding space reduces to roughly 5 effective parameters
via dimension reduction, alongside a linear-separability finding on the
same embeddings (see Linear Separability). See
el-samman-etal-2024-a-real-schnet-family-gnns-128-parameter-qm9-molecular-embedding-space-reduces-to-around-5-effective-parameters-with-sharp-linear-boundaries-separating-chemical-moieties.
Extension to brain alignment, and a causal fine-tuning intervention
Cheng, Vaidya & Antonello (2026) estimate per-layer intrinsic dimension
(via GRIDE) in real OPT (125M-13B), Pythia (160M-6.9B), WavLM
(base-plus, large) and Whisper (large) activations, finding layerwise
ID correlates strongly with how well that layer’s activations predict
real human brain responses (fMRI rho=0.76; ECoG rho=0.43), with the ID
peak layer and the best-brain-predicting layer usually 0-1 layers
apart. Causal validation: fine-tuning the best-performing WavLM layer
to better predict real fMRI responses (“brain-tuning”) causally raises
both encoding performance and that layer’s own measured intrinsic
dimension, with a random-Fourier-features negative control showing
raised ID alone does not suffice to cause the effect — a genuine causal
manipulation of the ID profile itself, distinct from the purely
observational depth-wise measurements elsewhere in this node. See
layerwise-intrinsic-dimension-of-real-opt-pythia-wavlm-and-whisper-activations-peaks-near-the-layer-that-best-predicts-real-fmri-ecog-brain-responses-and-brain-tuning-a-layer-causally-raises-both-its-id-and-its-brain-alignment.
Extension to a non-language, non-vision domain: single-cell transcriptomics
Kendiukhov (2026) extends the “compression coincides with peak semantic
content” pattern to a genuinely new domain and a new spectral measure:
across scGPT’s 12 transformer layers, the SVD effective rank of the
gene-embedding matrix collapses monotonically 23.6 to 1.6 (Spearman
rho=-1.000), with the top singular vector’s variance fraction rising
from 53.7% to 93.4% and TwoNN intrinsic dimensionality falling 32.6 to
18.1. Unlike the ESM-2/iGPT case (a local ID minimum partway through
the network, followed by re-expansion), this is a monotonic collapse to
the final layer — but the same qualitative claim holds: the most
spectrally-compressed layer’s low-rank subspace is the one that best
correlates with independent ground truth (STRING protein-protein-
interaction confidence, TRRUST transcription-factor-target relations,
cell-type identity), with a feature-shuffle control confirming the
collapse is not an artifact. See
scgpt-gene-embedding-effective-rank-collapses-14-fold-across-layers-and-the-final-compressed-layer-encodes-ppi-tf-target-and-cell-type-ground-truth.
Extension to whole-sequence shape space, rather than pointwise activations
Beshkov & Malthe-Sørenssen (2026) apply a categorically different notion
of dimensionality to the same expansion-then-compression question: rather
than TwoNN on pointwise activations, they treat each protein’s whole
per-residue PLM representation as a curve, map it to its square-root-
velocity (SRV) shape via SRV shape-space Fréchet radius and tangent-PCA effective dimension, and measure the
tangent-PCA effective dimension of the resulting shape space across
layers of ESM2 (35M/150M/650M/3B) and Ankh. The same qualitative
expansion-then-compression trajectory recurs — larger models expand the
shape-space dimension more sharply in early layers before contracting —
but at a far lower absolute dimension than PCA run directly on the same
models’ flattened pointwise activations, indicating that while individual
residue activations occupy a high-dimensional ambient space, the ways in
which different protein shapes (whole trajectories) differ from each
other are describable by only a handful of directions. A companion
graph-filtration analysis on the same models finds three-dimensional
protein structure is most faithfully encoded at very short (2-residue)
and moderate (8-residue) context lengths, and closer to but before the
final layer — independent evidence that the most structurally-faithful
layer is not the last one. See
beshkov-malthe-sorenssen-2026-plm-shape-spaces-undergo-the-same-expansion-then-compression-effective-dimension-trajectory-as-pointwise-activations-but-at-a-far-lower-absolute-dimension.
A conflicting depth trend, not yet reconciled
Cai, Huang, Bian & Church (2021) estimate a related but distinct local-dimension statistic — Local Intrinsic Dimension (LID), via a -nearest-neighbor expansion-model estimator rather than TwoNN — across BERT, DistilBERT, GPT, GPT-2, and ELMo, and find LID increases nearly linearly with depth in every model, the opposite trajectory shape from the expansion-then-compression pattern documented above. Both findings stand as verified in this map; the discrepancy may reflect a genuine difference between language models and the protein/image models studied here, a genuine difference between the LID and TwoNN estimators, or both — flagged as an open question rather than resolved. See manifold-dimension-lower-than-ambient.
A theoretical impossibility casts doubt on the expansion-then-compression shape itself
Schulte & Rügamer (2026) prove that because standard network layers (linear/conv, ReLU, softmax, pooling, residual connections, BatchNorm/RMSNorm, and their compositions including self-attention) are Lipschitz maps, both pointwise and Hausdorff intrinsic dimension can only stay the same or decrease from one layer to the next — they can never rise. Yet reproducing the standard TwoNN/MLE/Gride pipeline on ResNet-34 (ImageNet) and on Llama-3.1-8B, Mistral-7B-v0.3, and Pythia-6.9B (WikiText prompts, last-hidden-state per layer) recovers the familiar hump-shaped early-expansion-then-compression pattern documented elsewhere in this node — a pattern their own theorem says is mathematically impossible for the true ID to exhibit. Rather than proposing a replacement estimator, they investigate what these estimators are actually tracking instead (nearest-neighbor distances, ambient dimension, cosine similarity, representation norm, entropy), concluding that the widely-cited “abstraction emerges as a mid-layer dimensionality peak” narrative built on TwoNN-style estimators is very likely an artifact of estimator bias rather than a real geometric signal. This is a direct methodological caution for every expansion-then-compression finding in this node that relies on TwoNN or closely related nearest-neighbor-ratio estimators (though not for findings, such as MST-based or spectral-entropy/effective-rank measures, that use different machinery). See lipschitz-layers-cannot-increase-true-intrinsic-dimension-so-the-standard-twonn-hump-shaped-id-profile-is-likely-an-estimator-artifact.
Relative to tangent-aligned-anisotropy
Tangent-Aligned Anisotropy Hypothesis proposes a mechanism for why representations end up occupying a low-dimensional subspace in the first place — frequency-concentrated sampling and self-reinforcing tangent-aligned gradients — rather than measuring the resulting dimensionality profile directly the way this concept does. The two are complementary: this concept characterizes what the dimensionality trajectory looks like across depth; that hypothesis proposes why training would produce low-dimensional, anisotropic representations at all.
Relative to sufficiency-staging
Attention–MLP Sufficiency Staging Hypothesis proposes a sharper, mechanistically-located version of the same kind of compression trajectory: rather than a generic expansion-then-plateau across many layers, it claims a two-stage split tied to one specific sublayer boundary (attention output vs. post-MLP), where what’s being compressed is a known, analytically-derivable sufficient statistic rather than “intrinsic dimension” in the abstract. This concept describes the general shape of the trajectory; that hypothesis describes one specific, more tightly-scoped mechanism producing a compression step within it.
Token-level local ID as a per-token, rather than per-layer, statistic
Lee, Weber, Viegas & Wattenberg (2025) apply a k-NN-neighborhood, PCA-based local intrinsic-dimension estimator (components needed for 95% variance) to individual tokens’ embeddings — not layer-by-layer activations of a shared input — across GPT2, Llama3, Gemma2, GPT-NeoX-20B and OLMo-7B, finding that low-ID tokens (GPT2-medium range 508-635, versus a random-Gaussian baseline of ~605-611) form semantically coherent clusters while high-ID tokens do not. This is a within-layer, across-vocabulary application of the same local-dimension-estimation toolkit used elsewhere in this map for across-depth profiles, showing the estimator also picks out meaningful structure when applied to a fixed layer’s full token population rather than a fixed input’s trajectory across layers. See lee-etal-2025-cross-model-embedding-orientation-similarity-drops-across-families-local-id-clusters-tokens-linear-map-transfers-steering-vectors.
Per-concept local ID as a complexity proxy, rather than a per-token or per-layer statistic
Skierś, Trzciński & Deja (2026, ELROND) apply local intrinsic-dimension estimation to a third axis beyond depth or token identity: individual concepts. Backpropagating the differences between stochastic image realizations of the same text prompt in real SDXL, then decomposing the resulting gradient directions via PCA or a sparse autoencoder, they estimate the LID of each decomposed concept-direction’s own manifold within the text embedding and find general concepts (e.g. “Dog”) consistently have higher LID than specific hyponyms (e.g. “Poodle”), validated against WordNet. Causally, injecting these directions into a distilled, mode-collapsed student model (SDXL-DMD) restores output diversity toward the teacher’s distribution (measured via FID), tying the per-concept dimensionality measurement to a genuine generative-diversity effect. See gradient-derived-concept-directions-in-real-sdxl-and-sdxl-dmd-text-embeddings-have-local-intrinsic-dimension-tracking-concept-generality-and-injecting-them-causally-restores-fid-lost-to-distillation.
A training-dynamics phase transition dissociating linear from nonlinear dimension
Lee, Jiralerspong, Yu, Bengio & Cheng (2024) track nonlinear intrinsic dimension (TwoNN) and linear PCA effective dimension across pretraining on Pythia-410M/1.4B/6.9B, finding nonlinear intrinsic dimension undergoes a sharp phase transition at training step t~10^3 that coincides with the onset of zero-shot task competence, consistent across model sizes and generalizing to fully-trained Llama-3-8B and Mistral-7B. Linear PCA effective dimension, by contrast, correlates with superficial/Kolmogorov data complexity (gzip compressibility) rather than with this emergence event — a dissociation between the linear and nonlinear dimensionality measures themselves, orthogonal to (and a useful complement to) this node’s own expansion-then-compression depth trajectory, since it concerns training dynamics at fixed depth rather than a depth trajectory at fixed training time. See lee-etal-2024-nonlinear-intrinsic-dimension-shows-a-sharp-phase-transition-coinciding-with-zero-shot-task-competence-while-linear-pca-dimension-tracks-only-superficial-complexity.
Local intrinsic dimension predicts LLM truthfulness with a hunchback layer profile
Yin, Srinivasa & Chang (2024) compute Local Intrinsic Dimension (LID, via
GeoMLE) on the per-layer activations of real Llama-2-7B and Llama-2-13B
generating answers on four real QA datasets (TriviaQA, HotpotQA,
TydiQA-GP, CoQA), finding LID traces a hunchback shape across layers —
rising, peaking mid-network, then falling — that closely tracks (shifted
one to two layers behind) the layer-wise truthfulness-detection AUROC,
peaking at 0.746 and outperforming entropy- and classifier-based
baselines; the LID-truthfulness relationship transfers with only modest
degradation across QA datasets. See
yin-etal-2024-the-local-intrinsic-dimension-of-real-llama-2-activations-traces-a-hunchback-shape-across-layers-that-predicts-generation-truthfulness-on-real-qa-datasets.
Local dimension of fine-tuned representations tracks task success and overfitting onset
Ruppik, von Rohrscheidt, van Niekerk, Heck, Vukovic, Feng, Lin, Lubis,
Rieck, Zibrowius & Gasic (2025) apply the TwoNN local-dimension estimator
to real RoBERTa-base embeddings, comparing the base masked-LM checkpoint
against real task fine-tunes: a TripPy-R dialogue-state tracker
fine-tuned on real MultiWOZ 2.1, and a separate fine-tune on real EmoWOZ
7-class emotion recognition. Local dimension drops markedly for the
MultiWOZ fine-tune relative to the base model, coinciding (in an
auxiliary synthetic grokking task used to validate the estimator) with
the onset of rising validation accuracy; on the real EmoWOZ fine-tune,
local dimension instead rises starting at the same epoch validation loss
begins to increase, tying rising local dimension to overfitting onset
rather than task mastery. See
ruppik-etal-2025-local-dimension-of-real-fine-tuned-roberta-embeddings-drops-with-successful-dialogue-state-tracking-and-rises-with-overfitting-onset-on-real-emotion-recognition-fine-tuning.
How to detect it
Estimate ID layer-by-layer with a neighbor-distance-ratio estimator (TwoNN or similar), plot it against relative depth, and check for a peak-then-compression shape; cross-check that a semantic-overlap measure (agreement with ground-truth labels among each point’s nearest neighbors) is maximized at or near the ID’s local minimum rather than at the final layer.
Memorized sequences leave a local low-dimensionality signature
Arnold (2025) estimates local intrinsic dimension of hidden-state representations across GPT-Neo (125M/1.3B/2.7B) and BERT while replaying training sequences, finding sequences the model has memorized verbatim show a systematically lower local ID in mid-to-late layers than non-memorized sequences of matched surface statistics, with the gap widening with model scale. A simple ID-based scoring rule detects memorized sequences at accuracy competitive with loss-based membership-inference baselines — extending this node’s central claim (dimensionality profile correlates with what a layer’s representation is “doing”) to a training-data-property axis (memorized vs. not) rather than a depth or training-step axis. See intrinsic-dimension-tracks-memorization-in-language-models.
Effective rank of multi-response, multi-layer embeddings flags hallucination
Wang, Wei, Yue & Sun (2025) directly compute the effective rank (Roy &
Vetterli 2007 spectral-entropy dimensionality measure) of a matrix
formed by concatenating embeddings sampled across multiple generated
responses and multiple layers of Llama-2-7b-chat, Llama-2-13b-chat, and
Mistral-7B-v0.1, and use it as a hallucination-detection signal,
reaching AUROC around 0.84-0.86 across BioASQ and other QA benchmarks —
competitive with or exceeding semantic-entropy and self-consistency
baselines. See the-effective-rank-of-multi-response-multi-layer-embedding-matrices-detects-llm-hallucination-at-auroc-0-84-to-0-86-competitive-with-semantic-entropy-baselines.
Combined with anisotropy for hallucination detection
Srey, Wu, Nguyen & Luu (2026) pair a log-pseudo-determinant-of-covariance
dimensionality measure with a circular-variance anisotropy measure
(see anisotropy) per generated token across Llama-3.2-3B-Instruct,
Llama-3.1-8B-Instruct, and Ministral-8B-Instruct, beating prior
uncertainty baselines for hallucination detection across seven
benchmarks. See per-token-log-pseudo-determinant-of-cross-layer-covariance-and-circular-variance-of-hidden-states-detect-llm-hallucination-better-than-prior-uncertainty-baselines.
Relative layer-depth of a fixed probe direction shifts with model scale
Manek (2026) extends a diff-of-means evaluation-awareness probe across 11
open-weight models spanning three families (Qwen2.5 0.5B-32B, Gemma2
2B/9B/27B, Llama-3.2 1B/3B), measuring per-layer AUROC on a held-out
oversight-detection split rather than a raw dimensionality estimator. The
relative depth (layer / total layers) at which decoding AUROC peaks
shifts systematically with scale within a family: Qwen2.5 1.5B/3B peak
very late (relative depth 0.96-0.97) while 14B/32B peak at the earliest
layers (0.021-0.031); Gemma2 shows the same late-to-early shift
(0.885 to 0.304) from 2B to 27B; Llama-3.2 stays mid-layer at both sizes
tested. This is a different kind of depth-profile claim than the
TwoNN/effective-rank trajectories elsewhere in this node — not “how does
a layer’s own dimensionality change with depth” but “at what relative
depth does a fixed semantic direction become most linearly decodable,
and how does that depth itself move as a function of scale” — but it
shares the same underlying phenomenon that a network’s geometrically
and functionally relevant layer is not fixed at a constant relative
position across model sizes. See
the-layer-depth-at-which-a-diff-of-means-evaluation-awareness-direction-peaks-in-decoding-accuracy-shifts-from-late-layers-in-small-models-to-the-earliest-layers-in-large-ones-within-the-same-family.
Per-head effective rank tracked across training, rather than across depth
Xu (2026) applies the same effective-rank instinct that motivates this
node’s TwoNN/spectral-entropy measures, but to a different axis
entirely: rather than tracking overall representation dimensionality
across network depth, this work computes the participation ratio
(spectral-entropy effective rank) of each individual attention head’s
own per-token output activation matrix, integrated across training
time. The resulting per-head spectral signal, thresholded via a
task-pattern screen and confirmed by causal ablation, identifies which
heads are doing specialized circuit computation without any behavioral
labels — validated across 7 model configurations spanning a 51M-7B
parameter range (dense and MoE architectures, four different
pretraining corpora), with the fraction of heads doing identifiable
specialized computation conserved at roughly 17-19% across the full
scale range, and the signal correctly recovering each of 6 different
pretraining seeds’ entirely distinct circuit head-sets without labels.
See
participation-ratio-spectral-signal-identifies-idiosyncratic-per-seed-attention-head-circuits-without-labels-and-matches-a-conserved-fraction-of-specialized-heads-across-an-8x-scale-range.
A cross-model convergence claim paired with a depth-wise scale trajectory, in a non-language domain
Craig, Selz, Beylich & Tempest (2026) find, alongside a CKA cross-model
convergence claim (see Platonic Representation Hypothesis), that GraphCast and
Aurora’s processor layers show a consistent depth-wise trajectory:
large-spatial-scale changes dominate in early layers, shifting to
smaller-spatial-scale changes with increasing depth. Framed via a
“particle description” hypothesis (latent variables as particle
positions moving under gradient flow toward a learned free-energy
minimum), this is a depth-wise structural trajectory documented in AI
weather models rather than the language/vision/protein transformers
studied elsewhere in this node. See
graphcast-and-aurora-show-similar-cka-representational-geometry-with-a-depth-wise-shift-from-large-to-small-spatial-scale-changes.
An information-theoretic effective-features count, cross-checked against prior intrinsic-dimensionality studies
Bereska, Tzifa-Kratira, Samavi & Gavves (2025) define an information-
theoretic “effective features” metric (the exponential
of the Shannon entropy of SAE feature activation magnitudes) and a
superposition ratio . Layer-wise effective-feature patterns
computed on real pretrained Pythia-70M activations mirror this node’s
prior intrinsic-dimensionality findings, and the same metric captures a
sharp feature-consolidation transition during grokking. See
an-information-theoretic-effective-features-count-from-sae-activation-entropy-tracks-prior-intrinsic-dimensionality-findings-layer-by-layer-in-pythia-70m.
An MST-based estimator, replacing linear-probe accuracy as a representation-quality proxy
Mordacq, Kalogeiton & Oudot (2026) estimate intrinsic dimension of frozen
penultimate-layer representations from 33 pretrained SSL vision models —
ResNet-50 and ViT-S/B/L/G backbones across joint-embedding (VICReg,
DINO), joint-predictive (I-JEPA), combined (iBOT, DINOv2), and
vision-language (CLIP, EVA-CLIP) objectives — using a minimum-spanning-
tree estimator (dim_MST) rather than the nearest-neighbor-ratio TwoNN
estimator used elsewhere in this node. Across ImageNet, iNat-18/21,
CIFAR-10/100 and SUN397, dim_MST strongly and consistently correlates
with downstream linear-probe accuracy, positioning intrinsic dimension as
a cheap proxy for representation quality that doesn’t require training a
probe at all. See
mst-intrinsic-dimension-of-frozen-ssl-vision-representations-correlates-with-downstream-linear-probe-accuracy-across-33-models.
Local intrinsic dimensionality as a per-input anomaly signature, not a global quality number
Arcos-Holzinger, Erfani, Bailey & Khudanpur (2026) turn intrinsic-
dimension estimation into a diagnostic tool: computing per-layer Local
Intrinsic Dimensionality (LID) on WavLM and wav2vec 2.0 representations
under acoustic perturbation, they find benign low-SNR noise’s LID profile
converges back toward the clean-input profile as SNR increases, while
adversarial perturbations retain elevated LID in early layers regardless
of SNR — a geometric signature distinguishing adversarial from benign
inputs (AUROC 0.78-1.00) without needing ground-truth transcripts. See
local-intrinsic-dimensionality-of-wavlm-and-wav2vec2-representations-diverges-between-benign-noise-and-adversarial-perturbations-enabling-transcript-free-anomaly-detection.
Prompt-dependent intrinsic dimension in a diffusion model, layer-dependent in its own right
Kvinge, Brown & Godfrey (2023) measure intrinsic dimension of Stable
Diffusion’s internal representations at specified bottleneck and latent
layers, across denoising steps and varying prompts, finding prompt
choice substantially affects the measured dimension. The effect is
itself layer-dependent: in certain bottleneck layers, intrinsic dimension
correlates with prompt perplexity (via a surrogate language model), while
this correlation vanishes in the latent layers — an early (2023),
foundational instance of this node’s cross-modality intrinsic-dimension
program applied to a generative diffusion model rather than a
discriminative or self-supervised one. See
stable-diffusions-internal-representation-intrinsic-dimension-depends-on-prompt-and-correlates-with-prompt-perplexity-in-bottleneck-but-not-latent-layers.
Density-peak cluster geometry distinguishes in-context learning from fine-tuning within one LLM
Doimo, Serra, Ansuini & Cazzaniga (2024) combine intrinsic-dimension
estimation with Density-peak clustering (Advanced Density Peak) on Llama3-8B’s last-token
hidden representations, comparing in-context learning (ICL) and
supervised fine-tuning (SFT) layer by layer. Both regimes undergo a
sharp two-phase transition around layer 17, marked by a peak in intrinsic
dimension and a jump in cluster count/geometry; before the transition,
ICL organizes representations into far more (60-70 vs. under 40) and more
sharply-separated (core-point fraction ~0.6) semantic clusters than SFT,
while after it, SFT instead develops sharper probability modes encoding
answer identity — evidence that two training regimes applied to the same
base model induce measurably different discovered manifold geometries,
not just different downstream accuracy. See
icl-and-fine-tuning-in-llama3-8b-undergo-a-shared-layer-17-phase-transition-but-produce-measurably-different-density-peak-cluster-counts-and-separation-before-it.
In-context learning occupies a consistently higher-ID regime than fine-tuning, dissociated from task accuracy
Janapati & Ji (2024) compare in-context learning (ICL), LoRA supervised
fine-tuning (SFT), and zero-shot on real Llama-3-8B, Llama-2-13B,
Llama-2-7B, and Mistral-7B-v0.3 across 8 tasks, using TwoNN on
last-token hidden states. ICL with 5 or more demonstrations induces a
consistently higher intrinsic-dimension profile across all layers than
SFT or zero-shot, even where SFT achieves higher task accuracy (e.g.
MMLU: ICL-10 accuracy 0.531 vs. SFT 0.542) — a scale-and-checkpoint
replication of the general finding that different learning paradigms
applied to the same base model leave measurably different dimensionality
footprints (compare the ICL-vs-SFT cluster-geometry finding above), with
ID-vs-demonstration-count itself non-monotonic, rising then
plateauing/decreasing past k~5-10. See
janapati-ji-2024-in-context-learning-induces-consistently-higher-intrinsic-dimension-than-supervised-fine-tuning-across-real-llama-and-mistral-checkpoints-even-with-lower-task-accuracy.
An early, foundational training-time ID trajectory (expansion then compression)
Razzhigaev, Mikhalchuk, Goncharova, Oseledets, Dimitrov & Kuznetsov
(2024) track TwoNN intrinsic dimension across real pretraining
checkpoints of Bloom-3B and Pythia-2.8B rather than across depth: “the
intrinsic dimension of embeddings increases in the initial phases of
training… followed by a compression phase towards the end of
training with dimensionality decrease” — an early, foundational
instance of the same expansion-then-compression shape this node’s
depth-wise entries document, but along the training-time axis (see
also the later, more mechanistically detailed Lee et al. 2024
training-dynamics phase-transition entry above). Purely observational.
See razzhigaev-etal-2024-embedding-intrinsic-dimension-expands-in-early-pretraining-then-compresses-toward-the-end-tracked-via-twonn-across-bloom-3b-and-pythia-2-8b-checkpoints.
A global-to-local dimension ratio tracks manifold untangling across a reasoning trajectory and model scale
Anderson (2026) pairs PCA global effective dimension () with the
Levina-Bickel MLE local intrinsic dimension () along
chain-of-thought reasoning trajectories in real Llama-3-8B-Instruct and
Llama-3.1-70B-Instruct, defining their ratio (Global-to-Local Dimension
Ratio, G/L) as an “untangling” statistic distinct from this node’s usual
per-layer-depth trajectories: here the axis is generation-time (CoT-step)
position, compared across model scale rather than network depth at fixed
input. Law-reasoning trajectories show a 10x untangling effect (G/L
9.82x->0.98x) between the two scales. Single-author preprint;
observational only. See
anderson-2026-a-global-to-local-dimension-ratio-tracks-manifold-untangling-across-cot-reasoning-trajectories-and-model-scale.
Key papers
- Ansuini, Laio, Macke & Zoccolan (2019). Intrinsic Dimension of Data Representations in Deep Neural Networks. — origin of the TwoNN-based ID-profile analysis, in convolutional (non-transformer) networks.
- Valeriani, Doimo, Cuturello, Laio, Ansuini & Cazzaniga (2023). The Geometry of Hidden Representations of Large Transformer Models. NeurIPS 2023, arXiv:2302.00294 — extends the analysis to transformers (ESM-2, iGPT), links the ID minimum to peak semantic content, and reports a preliminary Llama-2-70B result.
- Choi, J., Hwang, G., Cho, H. & Kang, M. (2023). Analyzing the Latent Space of GAN through Local Dimension Estimation. arXiv:2205.13182 — spatially-varying local ID over a real StyleGAN2 latent manifold, correlated with disentanglement.
- El-Samman, A. M., Husain, I. A., Huynh, M., De Castro, S., Morton, B. & De Baerdemacker, S. (2024). Global geometry of chemical graph neural network representations in terms of chemical moieties. Digital Discovery 3, 544-557 — low effective dimensionality of a real SchNet-family GNN’s QM9 molecular embeddings.
- Yin, F., Srinivasa, J. & Chang, K.-W. (2024). Characterizing Truthfulness in Large Language Model Generations with Local Intrinsic Dimension. ICML 2024, arXiv:2402.18048 — LID hunchback layer profile predicting truthfulness in real Llama-2-7B/13B.
- Ruppik, B. M., von Rohrscheidt, J., van Niekerk, C., Heck, M., Vukovic, R., Feng, S., Lin, H., Lubis, N., Rieck, B., Zibrowius, M. & Gasic, M. (2025). Less is More: Local Intrinsic Dimensions of Contextual Language Models. NeurIPS 2025, arXiv:2506.01034 — local dimension of fine-tuned RoBERTa tracking task success and overfitting onset.