Definition

A manifold of dimension $n$ is a topological space $M$ in which every point $p \in M$ has an open neighbourhood homeomorphic to an open ball in $\mathbb{R}^n$. A pair (neighbourhood, homeomorphism) is a chart; a collection of charts covering $M$ is an atlas. A Riemannian manifold additionally carries a smoothly varying inner product $g_p$ on each tangent space $T_pM$, which lets lengths, angles, and geodesics be measured intrinsically.

The activation space of a network is $\mathbb{R}^d$, but the set of activations a network actually produces need not fill $\mathbb{R}^d$ — it can concentrate on a lower-dimensional manifold $M \subset \mathbb{R}^d$.

Intuition

Earth is a 2-dimensional manifold: locally, around any point, it looks like a flat plane (a city map is a chart), yet globally it is a sphere — no single flat map can cover it without distortion or seams. A manifold is exactly this local-flat/global-curved compromise, made precise.

Properties

The classification question for a manifold found in activation space is: what is its curvature, and is it open or closed? These two questions separate the shapes this map catalogues:

Type	Curvature	Topology	$\dim$ (typical)	$H_1$
Circle	0	closed, periodic	1	$\mathbb{Z}$
Torus	0	closed, periodic, product of two circles	2	$\mathbb{Z}^2$
Sphere	$+1$	closed, simply connected	$n$	$0$
Hyperbolic Manifold	$-1$	open	$n$	$0$
Affine Subspace	0	open, flat	$k$	$0$
1D continuum manifold	0 (extrinsic ripples)	open, non-periodic	1	$0$
Paraboloid (circular × pinched continuum)	varies (circle radius depends on axial coordinate)	open (in axial coordinate) × closed (circular factor)	2	$\mathbb{Z}$ (from the circular factor, away from any pinch point)

Curvature and topology are logically independent axes — flat and closed ($T^2$), flat and open (affine), positively curved and closed ($S^n$), negatively curved and open (hyperbolic) all occur. First homology $H_1$ (the free abelian group of independent non-contractible loops) is what discriminates a closed periodic manifold from an open one, and a circle from a torus.

A related but distinct question, orthogonal to the curvature/topology table above, is not “what shape is the manifold at one layer” but “how does the manifold’s own dimensionality change across depth” — see Intrinsic-dimension profile across depth for the expansion-then-compression trajectory found so far in every self-supervised transformer studied.

A separate, non-Riemannian family of shapes catalogued here is built from bounded convex pieces rather than curvature: Polytope (Simplex) for a single categorical concept’s convex hull, and Minkowski Sum of Tile Polytopes (Minkowski Representation Hypothesis) for a whole layer’s activation space as the vector sum of many such tile-local convex hulls.

A further distinction, orthogonal to both tables above, is between a manifold as a static description of where activations sit and a manifold as a dynamical claim about a recurrent network’s own update rule — see Line Attractor for a 1D manifold of fixed points with stability properties, not just a traced-out curve, and Limit cycle (stable periodic attractor) for its closed-orbit sibling: a contractive periodic attractor a network’s dynamics settle into and recover to, rather than a static loop traced by activations across inputs.

A chart only guarantees local homeomorphism to $\mathbb{R}^n$ — it says nothing about global shape. This is precisely why a manifold can be “locally boring” (looks flat close-up) yet globally have any of the topologies in the table above.

A fourth distinction, orthogonal to all of the above, is between a manifold as a description of where activations sit and a manifold pinched at a finite set of singular points where several senses of the same symbol are glued together — see Pinched Manifold (Singular Quotient) for Jakubowski, Gasić & Zibrowius (2020)‘s word-embedding singularity/polysemy measure, a genuine shape claim (not a measurement) with persistent homology used only as the detection instrument.

A related but separate question about decodability rather than shape: a world-state can be reliably recovered from activations by a nonlinear probe without being linearly separable in that basis — see Nonlinear World-Model Decodability for Li, Hopkins, Bau, Viégas, Pfister & Wattenberg (2023)‘s original Othello-GPT board-state finding (later reframed as linear by Nanda et al. 2023, already in this map) and its cross-architecture Procrustes-alignment convergence follow-up, Yuan & Søgaard (2025) — the latter now classified as evidence for Platonic Representation Hypothesis rather than a manifolds shape claim, since its central result is cross-model geometric convergence, not a single model’s shape.

A fifth, non-Riemannian question is not about a static shape at all but about a labeled, attribute-indexed deformation field on the embedding — see Attribute-Induced Embedding Folding for Leroy, Mastropietro, Nurisso & Vaccarino (2025)‘s multiscale face-recognition finding, where invariance to an attribute is achieved by nonlinearly folding the space along that attribute’s direction rather than by a fixed curvature or shape.

A related but distinct question, orthogonal to the tables above, is not “what shape is the manifold” but “how curved is it” — extrinsic principal curvature (MAPC, via local-PCA fitting) vs. intrinsic graph-Ricci curvature (Ollivier/Forman) are different, non-interchangeable quantities, tracked as a depth/training profile rather than a named shape. See Curvature profile of the representation manifold for both, including Kaufman & Azencot (2023)‘s MAPC-vs-generalization finding and Hehl, von Renesse & Weber (2025)‘s Ricci-flow-like contraction/reversal finding.

A third, similarly orthogonal question is not “what shape” or “how curved” but “what topological features (holes, loops, connected components) persist” in the activation point-cloud — a computed persistent-homology/Betti profile, not a named shape. See Persistent-homology / Betti profile across depth, including Gardinazzi et al. (2024)‘s layer-pruning application, Magai & Ayzenberg (2022)‘s joint topology/dimension-vs-generalization finding, and Kendiukhov (2026)‘s large-scale cross-model biological-foundation-model audit.

Key papers

• Karczewski, Heinonen, Pouplin, Hauberg & Garg (2026). The Spacetime of Diffusion Models: An Information Geometry Perspective. arXiv:2505.17517, ICLR 2026 Oral — proposes a distinct kind of diffusion-model geometry from either of the Jacobian-pullback approaches above: a Fisher-Rao information metric built directly on the exponential family of denoising distributions $\{p(x_0|x_t,t)\}$, proving the standard Riemannian pullback metric collapses to trivial Euclidean interpolation whenever latent and data space share dimension. See Diffusion Spacetime Information Geometry for the full construction and its validation on a real EDM2 ImageNet-512 model and a custom molecular (Alanine Dipeptide) diffusion model.
• Saito & Matsubara (2025). Image Interpolation with Score-based Riemannian Metrics of Diffusion Models. arXiv:2504.20288 — derives a genuine Riemannian metric ($G_{x_t}=J_{x_t}^\top J_{x_t}$) directly from a real diffusion model’s own score-function Jacobian, rather than pulling back an assumed linear latent metric (contrast Linear Direction‘s entry for Park et al. 2023); uses the metric’s geodesics to guide image interpolation in real Stable Diffusion, outperforming Lerp/Slerp/NAO baselines on CLIP-IQA reality/noisiness while matching the best reconstruction error, and replicates the same construction on a small custom diffusion model trained on MNIST. See a-riemannian-metric-derived-directly-from-stable-diffusions-own-score-function-jacobian-not-pulled-back-from-any-assumed-linear-latent-and-geodesic-guided-interpolation-outperforms-lerp-slerp-baselines.

Exercises

Base

1. Is a circle $S^1$ a manifold? If so, of what dimension, and what does a chart around a point look like?

Solution

Yes, $S^1$ is a 1-dimensional manifold. Around any point $p \in S^1$, a small arc is homeomorphic to an open interval $(-\varepsilon,\varepsilon) \subset \mathbb{R}^1$ (e.g. via the angle coordinate $\theta$ restricted to that arc). No single chart covers all of $S^1$ (the circle is compact, an open interval is not), so at least two overlapping charts are needed for a full atlas.

2. Give an example of two manifolds with the same dimension and curvature $0$ but different topology.

Solution

An affine plane $\mathbb{R}^2$ (Affine Subspace) and a torus $T^2$ (Torus) are both 2-dimensional and flat (zero Gaussian curvature everywhere), but $\mathbb{R}^2$ is open and simply connected ($H_1 = 0$) while $T^2$ is closed with $H_1 = \mathbb{Z}^2$. Curvature alone does not determine topology.

Middle

3. Show that a manifold’s dimension is a well-defined, chart-independent quantity — i.e. that two charts around the same point must agree on $n$.

Solution

Suppose $\phi_1 : U \to \mathbb{R}^{n_1}$ and $\phi_2 : U \to \mathbb{R}^{n_2}$ are two charts on overlapping open neighbourhoods of $p$. The transition map $\phi_2 \circ \phi_1^{-1}$ is a homeomorphism between open subsets of $\mathbb{R}^{n_1}$ and $\mathbb{R}^{n_2}$. Invariance of domain (a theorem of algebraic topology) states that a homeomorphism between open subsets of $\mathbb{R}^{n_1}$ and $\mathbb{R}^{n_2}$ can only exist if $n_1 = n_2$. Hence dimension is well defined.

4. A network’s post-LayerNorm activations satisfy $\|\hat{x}\| = \text{const}$ for every input. What manifold do they lie on, and what is its dimension as a subset of $\mathbb{R}^d$?

Solution

They lie on a sphere $S^{d-1} \subset \mathbb{R}^d$ (a level set of the norm), which is a manifold of dimension $d-1$ — one less than the ambient space, since the norm constraint removes exactly one degree of freedom. See Sphere.

Pro

5. Two manifolds $M_1, M_2$ have $H_1(M_1) = \mathbb{Z}$ and $H_1(M_2) = \mathbb{Z}^2$. Can $M_1$ and $M_2$ be homeomorphic? Justify using a general topological fact, not just these two examples.

Solution

No. Homology groups are topological invariants: if $M_1 \cong M_2$ (homeomorphic), then $H_k(M_1) \cong H_k(M_2)$ for every $k$, since homology is a functor from topological spaces (and continuous maps) to abelian groups, and a homeomorphism is invertible, so it induces an isomorphism on homology. $\mathbb{Z} \not\cong \mathbb{Z}^2$ as abelian groups (e.g. $\mathbb{Z}^2$ has rank 2, $\mathbb{Z}$ has rank 1), so $M_1 \not\cong M_2$.

6. A researcher claims to have found a manifold in activation space with positive curvature at every sampled point but reports $H_1 \neq 0$. Is this possible for a closed 2-dimensional manifold? Use the Gauss–Bonnet theorem to justify your answer.

Solution

Not for a closed, connected, orientable 2-manifold with curvature bounded strictly away from $0$ (in fact strictly positive everywhere), if the surface is required to be compact without boundary: Gauss–Bonnet gives $\iint_M K \, dA = 2\pi\chi(M)$, where $\chi(M) = 2 - 2g$ for a genus-$g$ orientable closed surface. Strictly positive curvature everywhere forces $\iint_M K\,dA > 0$, hence $\chi(M) > 0$, which forces $g = 0$ (a topological sphere). A sphere has $H_1 = 0$. So strictly-positive curvature everywhere and $H_1 \neq 0$ (genus $\geq 1$, e.g. a torus) are incompatible for such a surface — the reported combination would indicate either a measurement error, a non-closed/non-orientable surface, or curvature that is not strictly positive at every point (only on average, or with sign changes not captured by a coarse sample).