Neural manifold capacity analysis

methods / Theoretical / Analytical / Neural manifold capacity analysis

Techniqueadvanced

Treats each class or category's set of activations as a geometric 'object manifold' and computes its capacity (how many such manifolds can be linearly separated per feature dimension), effective dimension (participation ratio), and radius directly from the manifold's covariance structure, plus inter-manifold correlation/axis-alignment statistics — a mean-field-theoretic generalization of linear separability from points to whole manifolds.

Used in (4 observations)

Concept manifolds transiently untangle right before a reasoning stepmeasured

structure: Polytope (Simplex) · models: Ministral 3 8B Reasoning, Qwen2.5-14B-Instruct, GPT-OSS-20B, Qwen2.5-7B-Instruct · paper: Emergent Manifold Separability during Reasoning in Large Language Models

DeepSpeech2 untangles word and phoneme manifolds while tangling speakermeasured

structure: Polytope (Simplex) · models: DeepSpeech2 (conv + batch-norm + RNN, CTC loss, trained on LibriSpeech) · paper: Untangling in Invariant Speech Recognition

Object-manifold radius and dimension track memorization and double descentmeasured

structure: Polytope (Simplex) · models: AlexNet (ImageNet image classifier, supervised), VGG (image classifier, various depths), ResNet (image classifier, various depths) · paper: On the Geometry of Generalization and Memorization in Deep Neural Networks

Prompting methods reorganize category-manifold geometry through distinct mechanismsmeasured

structure: Polytope (Simplex) · models: Llama-3.1-8B, Gemma-2-2B · paper: The Geometry of Prompting: Unveiling Distinct Mechanisms of Task Adaptation in Language Models