Intrinsic dimension estimation (TwoNN)

methods / Theoretical / Analytical / Intrinsic dimension estimation (TwoNN)

Techniqueintermediate

Estimates the local dimensionality of the manifold activations lie on directly from the ratio of each point's distances to its first and second nearest neighbors, without needing an explicit low-dimensional embedding.

Used in (22 observations)

AlphaEarth satellite embeddings occupy a low-dimensional, locally rotating manifoldmeasured

structure: Intrinsic-dimension profile across depth, Curvature profile of the representation manifold · models: AlphaEarth (satellite/Earth-observation foundation model) · paper: Characterizing AlphaEarth Embedding Geometry for Agentic Environmental Reasoning

A global-to-local PCA-dimension ratio tracks reasoning-trajectory untanglingmeasured

structure: Intrinsic-dimension profile across depth · models: Llama-3-8B-Instruct, Llama-3.1-70B-Instruct · paper: The Geometry of Thought: How Scale Restructures Reasoning in Large Language Models

CNN intrinsic dimension is hunchback-shaped; last-layer ID predicts accuracymeasured

structure: Intrinsic-dimension profile across depth · models: AlexNet (ImageNet image classifier, supervised), VGG (image classifier, various depths), ResNet (image classifier, various depths) · paper: Intrinsic Dimension of Data Representations in Deep Neural Networks

CosyVoice2's LM module has a clean composable emotion subspacemeasured

structure: Linear Subspace · models: CosyVoice2 · paper: A Geometric Perspective on Composable Emotion Steering in Text-to-Speech Models

Manifold curvature predicts sparse-autoencoder reconstruction-loss scaling floorsmeasured

structure: Intrinsic-dimension profile across depth · models: Gemma-2-2B, Gemma-2-9B · paper: The Geometric Wall: Manifold Structure Predicts Layerwise Sparse Autoencoder Scaling Laws

ICL and fine-tuning share a layer-17 transition but differ before itmeasured

structure: Intrinsic-dimension profile across depth · models: Llama-3-8B · paper: The Representation Landscape of Few-Shot Learning and Fine-Tuning in Large Language Models

Intrinsic dimension expands then compresses; semantics peak at the minimummeasured

structure: Intrinsic-dimension profile across depth · models: ESM-2 (35M), ESM-2 (650M), ESM-2 (3B), iGPT-S, iGPT-M, iGPT-L, Llama-2-70B, GPT-2-XL, Gemma-2-2B · paper: The Geometry of Hidden Representations of Large Transformer Models, The Geometry of Concepts: Sparse Autoencoder Feature Structure

Lower input-sequence intrinsic dimension predicts more verbatim memorizationmeasured

structure: Intrinsic-dimension profile across depth · models: GPT-Neo-125M, GPT-Neo-1.3B, GPT-Neo-2.7B, GPT-J-6B · paper: Memorization in Language Models through the Lens of Intrinsic Dimension

In-context learning induces higher intrinsic dimension than fine-tuningmeasured

structure: Intrinsic-dimension profile across depth · models: Llama-3-8B, Llama-2-13B, Llama-2-7B, Mistral-7B-v0.3 · paper: A Comparative Study of Learning Paradigms in Large Language Models via Intrinsic Dimension

Layer intrinsic-dimension peak aligns with the best brain-predicting layermeasured

structure: Intrinsic-dimension profile across depth · models: OPT-125M, OPT-1.3B, OPT-13B, Pythia-160M, Pythia-410M, Pythia-6.9B, WavLM-base-plus, WavLM Large, Whisper large · paper: Abstraction Induces the Brain Alignment of Language and Speech Models

Nonlinear intrinsic dimension phase-transitions with emergent zero-shot competencemeasured

structure: Intrinsic-dimension profile across depth · models: Pythia-410M, Pythia-1.4B, Pythia-6.9B, Llama-3-8B, Mistral-7B · paper: Geometric Signatures of Compositionality Across a Language Model's Lifetime

Token-embedding orientation is shared within families but drops across themmeasured

structure: Intrinsic-dimension profile across depth, Linear Direction · models: GPT-2-Medium, Llama-3.1-8B-Instruct, Gemma-2-9B-it · paper: Shared Global and Local Geometry of Language Model Embeddings

The hump-shaped ID profile is likely a TwoNN estimator artifactmeasured

structure: Intrinsic-dimension profile across depth · models: ResNet-34 (supervised, ImageNet), Llama-3.1-8B, Mistral-7B-v0.3, Pythia-6.9B · paper: Rethinking Intrinsic Dimension Estimation in Neural Representations

Topological descriptors (persistent homology) and intrinsic-dimension estimates of the last-layer data manifold in real trained CNNs change measurably across layers and training, and predict generalization performancemeasured

structure: Persistent-homology / Betti profile across depth · models: ResNet-18 (supervised, ImageNet) · paper: Topology and Geometry of Data Manifold in Deep Learning

Contextualized embeddings lie on a low-dimensional manifold rising with depthmeasured

structure: Intrinsic-dimension profile across depth · models: BERT-base-uncased, DistilBERT-base-uncased, GPT-1 (OpenAI GPT), GPT-2-small, ELMo (AllenNLP biLM, 1B Word Benchmark) · paper: Isotropy in the Contextual Embedding Space: Clusters and Manifolds

Decoder-only models peak anisotropy in middle layersmeasured

structure: Anisotropy · models: BERT-base-uncased, RoBERTa-base, ALBERT-base-v1, GPT-2-small, GPT-J-6B, OPT-13B, Llama-2-7B, Llama-2-7B-Chat, BLOOM-560M, BLOOM-3B, Pythia-2.8B, Falcon-7B, Falcon-7B-Instruct · paper: The Shape of Learning: Anisotropy and Intrinsic Dimensions in Transformer-Based Models

Embedding intrinsic dimension expands early in pretraining, then compressesmeasured

structure: Intrinsic-dimension profile across depth · models: BLOOM-3B, Pythia-2.8B · paper: The Shape of Learning: Anisotropy and Intrinsic Dimensions in Transformer-Based Models

Chain-of-thought trajectories collapse to under ten intrinsic dimensionsmeasured

structure: Intrinsic-dimension profile across depth · models: Qwen2.5-0.5B, Qwen2.5-1.5B, Qwen2.5-3B, Qwen-2.5-7B, Qwen2.5-14B, Qwen2.5-32B, Qwen2.5-72B, Qwen3-0.6B, Qwen3-1.7B, Qwen3-4B, Qwen3-8B, Qwen3-14B, Qwen3-32B, Gemma 3 1B, Gemma 3 4B, Gemma 3 12B, Gemma 3 27B, DeepSeek-R1-Distill-Qwen-1.5B, DeepSeek-R1-Distill-Qwen-14B, DeepSeek-R1-Distill-Qwen-32B · paper: Reasoning emerges from constrained inference manifolds in large language models

scGPT gene-embedding effective rank collapses 14-fold across layersmeasured

structure: Intrinsic-dimension profile across depth · models: scGPT (whole-human pretrained checkpoint) · paper: Multi-Dimensional Spectral Geometry of Biological Knowledge in Single-Cell Transformer Representations

Stable Diffusion's intrinsic dimension tracks prompt perplexity in bottleneck layersmeasured

structure: Intrinsic-dimension profile across depth · models: Stable Diffusion v1.4 · paper: Exploring the Representation Manifolds of Stable Diffusion Through the Lens of Intrinsic Dimension

Persistent-homology H1 cycle strength rises sharply at grokkingmeasured

structure: Circle · models: Grokking Modular-Arithmetic Transformer (2 layers, 4 heads, pre-LN, d_model=128, mod 113/149/197), Grokking Modular-Arithmetic MLP (3 hidden layers, width 512, d_embed=128, mod 113/149/197) · paper: Topological Signatures of Grokking

Multi-response effective rank detects hallucination at AUROC 0.84-0.86measured

structure: Intrinsic-dimension profile across depth · models: Llama-2-7B-Chat, Llama-2-13B-Chat, Mistral-7B-v0.1 · paper: Revisiting Hallucination Detection with Effective Rank-based Uncertainty