Intrinsic dimension estimation (TwoNN)

methods / Theoretical / Analytical / Intrinsic dimension estimation (TwoNN)

Techniqueintermediate

Estimates the local dimensionality of the manifold activations lie on directly from the ratio of each point's distances to its first and second nearest neighbors, without needing an explicit low-dimensional embedding.

Used in (30 observations)

Across 28 open language models, the intrinsic dimension of hidden representations follows a layer-wise profile with a hunch in the middle layers during multiple-choice decision makingmeasured

structure: Intrinsic-dimension profile across depth · models: · paper: Geometry of Decision Making in Language Models

AlphaEarth satellite embeddings occupy a low-dimensional, locally rotating manifoldmeasured

structure: Intrinsic-dimension profile across depth, Curvature profile of the representation manifold · models: AlphaEarth (satellite/Earth-observation foundation model) · paper: Characterizing AlphaEarth Embedding Geometry for Agentic Environmental Reasoning

A global-to-local PCA-dimension ratio tracks reasoning-trajectory untanglingmeasured

structure: Intrinsic-dimension profile across depth · models: Llama-3-8B-Instruct, Llama-3.1-70B-Instruct · paper: The Geometry of Thought: How Scale Restructures Reasoning in Large Language Models

CNN intrinsic dimension is hunchback-shaped; last-layer ID predicts accuracymeasured

structure: Intrinsic-dimension profile across depth · models: AlexNet (ImageNet image classifier, supervised), VGG (image classifier, various depths), ResNet (image classifier, various depths) · paper: Intrinsic Dimension of Data Representations in Deep Neural Networks

CosyVoice2's LM module has a clean composable emotion subspacemeasured

structure: Linear Subspace · models: CosyVoice2 · paper: A Geometric Perspective on Composable Emotion Steering in Text-to-Speech Models

Manifold curvature predicts sparse-autoencoder reconstruction-loss scaling floorsmeasured

structure: Intrinsic-dimension profile across depth · models: Gemma-2-2B, Gemma-2-9B · paper: The Geometric Wall: Manifold Structure Predicts Layerwise Sparse Autoencoder Scaling Laws

ICL and fine-tuning share a layer-17 transition but differ before itmeasured

structure: Intrinsic-dimension profile across depth · models: Llama-3-8B · paper: The Representation Landscape of Few-Shot Learning and Fine-Tuning in Large Language Models

In LLaVA-NeXT and OmniFusion (both Mistral-7B-based), visual and text token representations differ in linearity, intrinsic dimension and anisotropy across layersmeasured

structure: Intrinsic-dimension profile across depth, Anisotropy · models: LLaVA-NeXT-7B, OmniFusion (Mistral-7B backbone) · paper: MLLM-Microscope: Unlocking Hidden Structure Within Multimodal Large Language Models

Intrinsic dimension expands then compresses; semantics peak at the minimummeasured

structure: Intrinsic-dimension profile across depth · models: ESM-2 (35M), ESM-2 (650M), ESM-2 (3B), iGPT-S, iGPT-M, iGPT-L, Llama-2-70B, GPT-2-XL, Gemma-2-2B · paper: The Geometry of Hidden Representations of Large Transformer Models, The Geometry of Concepts: Sparse Autoencoder Feature Structure

Lower input-sequence intrinsic dimension predicts more verbatim memorizationmeasured

structure: Intrinsic-dimension profile across depth · models: GPT-Neo-125M, GPT-Neo-1.3B, GPT-Neo-2.7B, GPT-J-6B · paper: Memorization in Language Models through the Lens of Intrinsic Dimension

In-context learning induces higher intrinsic dimension than fine-tuningmeasured

structure: Intrinsic-dimension profile across depth · models: Llama-3-8B, Llama-2-13B, Llama-2-7B, Mistral-7B-v0.3 · paper: A Comparative Study of Learning Paradigms in Large Language Models via Intrinsic Dimension

Layer intrinsic-dimension peak aligns with the best brain-predicting layermeasured

structure: Intrinsic-dimension profile across depth · models: OPT-125M, OPT-1.3B, OPT-13B, Pythia-160M, Pythia-410M, Pythia-6.9B, WavLM-base-plus, WavLM Large, Whisper large · paper: Abstraction Induces the Brain Alignment of Language and Speech Models

Nonlinear intrinsic dimension phase-transitions with emergent zero-shot competencemeasured

structure: Intrinsic-dimension profile across depth · models: Pythia-410M, Pythia-1.4B, Pythia-6.9B, Llama-3-8B, Mistral-7B · paper: Geometric Signatures of Compositionality Across a Language Model's Lifetime

Token-embedding orientation is shared within families but drops across themmeasured

structure: Intrinsic-dimension profile across depth, Linear Direction · models: GPT-2-Medium, Llama-3.1-8B-Instruct, Gemma-2-9B-it · paper: Shared Global and Local Geometry of Language Model Embeddings

The hump-shaped ID profile is likely a TwoNN estimator artifactmeasured

structure: Intrinsic-dimension profile across depth · models: ResNet-34 (supervised, ImageNet), Llama-3.1-8B, Mistral-7B-v0.3, Pythia-6.9B · paper: Rethinking Intrinsic Dimension Estimation in Neural Representations

Real pretrained transformers of varying scale share a nearly constant peak intrinsic dimension despite a wide range of ambient hidden sizesmeasured

structure: Intrinsic-dimension profile across depth · models: GPT-2 Small, GPT-2 XL, OPT-125M, OPT-1.3B, Pythia-1B, Pythia-160M · paper: Latent Semantic Manifolds in Large Language Models

Topological descriptors (persistent homology) and intrinsic-dimension estimates of the last-layer data manifold in real trained CNNs change measurably across layers and training, and predict generalization performancemeasured

structure: Persistent-homology / Betti profile across depth · models: ResNet-18 (supervised, ImageNet) · paper: Topology and Geometry of Data Manifold in Deep Learning

Contextualized embeddings lie on a low-dimensional manifold rising with depthmeasured

structure: Intrinsic-dimension profile across depth · models: BERT-base-uncased, DistilBERT-base-uncased, GPT-1 (OpenAI GPT), GPT-2-small, ELMo (AllenNLP biLM, 1B Word Benchmark) · paper: Isotropy in the Contextual Embedding Space: Clusters and Manifolds

Neural regression collapse coincides with last-layer feature intrinsic dimension below that of the targets (ID_H < ID_Y), which over-compresses and hurts generalization, while non-collapsed regression models keep ID_H > ID_Ymeasured

structure: Dimensional collapse · models: Andriopoulos et al. 2025 multivariate regression MLPs (control tasks and synthetic data) · paper: Geometric Analysis of Neural Regression Collapse via Intrinsic Dimension

SatCLIP Earth-representation intrinsic dimension is far below ambient size and increases with spatial resolutionmeasured

structure: Intrinsic-dimension profile across depth · models: SatCLIP-L10 (location encoder, Sentinel-2 contrastive; L=10 spherical-harmonic band limit), SatCLIP-L40 (location encoder, Sentinel-2 contrastive; L=40 spherical-harmonic band limit) · paper: Measuring the Intrinsic Dimension of Earth Representations

Decoder-only models peak anisotropy in middle layersmeasured

structure: Anisotropy · models: BERT-base-uncased, RoBERTa-base, ALBERT-base-v1, GPT-2-small, GPT-J-6B, OPT-13B, Llama-2-7B, Llama-2-7B-Chat, BLOOM-560M, BLOOM-3B, Pythia-2.8B, Falcon-7B, Falcon-7B-Instruct · paper: The Shape of Learning: Anisotropy and Intrinsic Dimensions in Transformer-Based Models

Embedding intrinsic dimension expands early in pretraining, then compressesmeasured

structure: Intrinsic-dimension profile across depth · models: BLOOM-3B, Pythia-2.8B · paper: The Shape of Learning: Anisotropy and Intrinsic Dimensions in Transformer-Based Models

Chain-of-thought trajectories collapse to under ten intrinsic dimensionsmeasured

structure: Intrinsic-dimension profile across depth · models: Qwen2.5-0.5B, Qwen2.5-1.5B, Qwen2.5-3B, Qwen-2.5-7B, Qwen2.5-14B, Qwen2.5-32B, Qwen2.5-72B, Qwen3-0.6B, Qwen3-1.7B, Qwen3-4B, Qwen3-8B, Qwen3-14B, Qwen3-32B, Gemma 3 1B, Gemma 3 4B, Gemma 3 12B, Gemma 3 27B, DeepSeek-R1-Distill-Qwen-1.5B, DeepSeek-R1-Distill-Qwen-14B, DeepSeek-R1-Distill-Qwen-32B · paper: Reasoning emerges from constrained inference manifolds in large language models

Class-manifold intrinsic dimension expands then compresses across depthmeasured

structure: Intrinsic-dimension profile across depth · models: DeepNet convolutional classifier (Fashion-MNIST, 90.96% test accuracy), ResNet classifier (CIFAR-10, 87.6% test accuracy) · paper: Dimensionality compression and expansion in Deep Neural Networks

scGPT gene-embedding effective rank collapses 14-fold across layersmeasured

structure: Intrinsic-dimension profile across depth · models: scGPT (whole-human pretrained checkpoint) · paper: Multi-Dimensional Spectral Geometry of Biological Knowledge in Single-Cell Transformer Representations

Stable Diffusion's intrinsic dimension tracks prompt perplexity in bottleneck layersmeasured

structure: Intrinsic-dimension profile across depth · models: Stable Diffusion v1.4 · paper: Exploring the Representation Manifolds of Stable Diffusion Through the Lens of Intrinsic Dimension

Persistent-homology H1 cycle strength rises sharply at grokkingmeasured

structure: Circle · models: Grokking Modular-Arithmetic Transformer (2 layers, 4 heads, pre-LN, d_model=128, mod 113/149/197), Grokking Modular-Arithmetic MLP (3 hidden layers, width 512, d_embed=128, mod 113/149/197) · paper: Topological Signatures of Grokking

Multi-response effective rank detects hallucination at AUROC 0.84-0.86measured

structure: Intrinsic-dimension profile across depth · models: Llama-2-7B-Chat, Llama-2-13B-Chat, Mistral-7B-v0.1 · paper: Revisiting Hallucination Detection with Effective Rank-based Uncertainty

The intrinsic dimension of linguistic datasets in OPT-350m, OPT-1.3b and OPT-6.7b predicts their information-theoretic coding length, and high compression predicts rapid adaptationmeasured

structure: Intrinsic-dimension profile across depth · models: OPT-350M, OPT-1.3B, OPT-6.7B · paper: Bridging Information-Theoretic and Geometric Compression in Language Models

Token-embedding intrinsic dimension is far below extrinsic dimension (about 13-25 for word2vec, GloVe and fastText; 25-122 for Pythia 14M-12B, redundancy 72-98 percent) and drops sharply early in Pythia trainingmeasured

structure: Intrinsic-dimension profile across depth · models: word2vec (Google News, 300d), GloVe (Wikipedia + Gigaword, uncased), FastText (wiki-news-subwords-300, 300d), Pythia-14M, Pythia-70M, Pythia-160M, Pythia-410M, Pythia-1B, Pythia-1.4B, Pythia-2.8B, Pythia-6.9B, Pythia-12B · paper: Measuring Intrinsic Dimension of Token Embeddings