Sphere

OBJECTgeometric-objectsubsetK:positivemanifoldbeginnerhow it's classified →

Closed homogeneous manifold of constant positive curvature. Normalisation onto a fixed norm places activations on the unit hypersphere Sⁿ⁻¹, making cosine similarity the natural distance.

Replicationcomputed from the corpus — never hand-assigned

4 papers · no shared authors2 architecture classes · across papers2 domains · across papers6 model families · across papers

Filled = two or more values reported by papers that share no author — replication. Outlined = two or more values, but all from a single study — breadth, not replication. Grey = a single value. Derived from paper authorship and each model's architecture class, domain and family; it updates itself when a paper is added.

Definition

The unit $n$ -sphere: $S^n = \{x \in \mathbb{R}^{n+1} : \|x\| = 1\}$ , an $n$ -dimensional manifold (dimension one less than the ambient space, since one degree of freedom is fixed by the norm constraint). For a $d$ -dimensional activation space, the relevant sphere is $S^{d-1} \subset \mathbb{R}^d$ .

Intuition

Constraining every point in a $d$ -dimensional space to have the same distance from the origin removes exactly one degree of freedom, leaving a $(d-1)$ -dimensional curved surface — the higher-dimensional analogue of how the surface of a globe is a curved 2D surface embedded in 3D space, even though every point on it is equally far from the center.

Properties

Constant positive curvature. $S^n$ has constant sectional curvature $+1/\rho^2$ for a sphere of radius $\rho$ (curvature $+1$ for the unit sphere), the unique simply-connected space form of constant positive curvature.
Simply connected for $n \geq 2$ . $\pi_1(S^n) = 0$ and $H_1(S^n) = 0$ for $n \geq 2$ — every loop can be contracted to a point. (The case $n=1$ , $S^1$ , is the exception — see Circle.)
Geodesics are great circles. The shortest path between two points on $S^n$ lies on the great circle through them (the intersection of $S^n$ with the 2-plane through both points and the origin). Geodesic distance is $\rho\cdot\arccos\!\big(\frac{x\cdot y}{\rho^2}\big)$ for points $x,y$ on a sphere of radius $\rho$ .
Normalisation projects onto $S^{d-1}$ . Any map of the form $x \mapsto \rho\, x/\|x\|$ sends all of $\mathbb{R}^d\setminus\{0\}$ onto the sphere of radius $\rho$ ; every point that shares a direction with $x$ maps to the same image.
Cosine similarity is the natural inner product on the sphere. For unit vectors $\hat x, \hat y \in S^{d-1}$ , $\hat x \cdot \hat y = \cos\alpha$ where $\alpha$ is the angle between them — equivalently, the geodesic distance is a monotonic function of cosine similarity, $d_{\text{geo}} = \arccos(\hat x\cdot\hat y)$ , so ranking by cosine similarity and ranking by geodesic distance agree exactly.
Volume concentration in high dimension. As $d \to \infty$ , the surface measure of $S^{d-1}$ concentrates increasingly sharply near any equator (a manifestation of measure concentration in high dimensions) — most of the sphere’s surface area lies within a thin band around any fixed great circle.

Key evidence

Wang & Isola (ICML 2020, arXiv:2005.10242) prove that contrastive representation learning’s success on the hypersphere decomposes exactly into two geometric quantities: an alignment loss (positive pairs close together) and a uniformity loss whose population-level minimizer is provably, via classical sphere-packing/potential theory, the uniform distribution $\sigma_d$ on $S^{d-1}$ itself — the strongest possible “no wasted volume” claim about a normalized representation’s distributional shape, beyond any single point’s norm or any pair’s angle. Training directly on the two decomposed losses, with no contrastive term, causally reproduces standard contrastive pretraining’s downstream performance (67.69% vs. 67.5% ImageNet linear-probe top-1, ResNet-50/MoCo v2). See contrastive-loss-asymptotically-optimizes-alignment-and-hypersphere-uniformity via Alignment/Uniformity Loss Decomposition.

Exercises

Base

If $x \in \mathbb{R}^d$ has $\|x\| = 5$ , what is $\hat x = x/\|x\|$ , and what is $\|\hat x\|$ ?

Solution

$\hat x = x/5$ , and $\|\hat x\| = \|x\|/5 = 5/5 = 1$ — normalisation always produces a unit-norm vector, by construction, regardless of the original norm (as long as it’s nonzero).

Two unit vectors satisfy $\hat x \cdot \hat y = 0$ . What is the angle between them, and what is their geodesic distance on $S^{d-1}$ (unit sphere)?

Solution

$\cos\alpha = 0 \Rightarrow \alpha = \pi/2$ (90°). Geodesic distance on the unit sphere is $d_{\text{geo}} = \arccos(0) = \pi/2$ .

Middle

Show that for unit vectors $\hat x, \hat y \in S^{d-1}$ , the Euclidean (chord) distance $\|\hat x - \hat y\|$ is a monotonically increasing function of the geodesic distance $\arccos(\hat x \cdot \hat y)$ , so that ranking points by chord distance and by geodesic distance agree.

Solution

$\|\hat x - \hat y\|^2 = \|\hat x\|^2 + \|\hat y\|^2 - 2\hat x\cdot\hat y = 2 - 2\cos\alpha$ , where $\alpha = \arccos(\hat x\cdot\hat y)$ is the geodesic distance. Since $\cos\alpha$ is strictly decreasing on $\alpha \in [0,\pi]$ , $2-2\cos\alpha$ is strictly increasing in $\alpha$ , and $\|\hat x-\hat y\| = \sqrt{2-2\cos\alpha}$ is a monotonic (increasing, since square root is monotonic on non-negative reals) function of $\alpha$ . Hence chord distance and geodesic distance induce the same ordering on any set of unit vectors.

A vector $x \in \mathbb{R}^d$ is decomposed as $x = \|x\|\,\hat x$ . Under the map $x \mapsto \hat x$ , show that all positive scalar multiples of $x$ (i.e. $\{\lambda x : \lambda > 0\}$ ) map to the same point on $S^{d-1}$ , and describe the fibers of this map geometrically.

Solution

For $\lambda > 0$ : $\widehat{\lambda x} = \lambda x / \|\lambda x\| = \lambda x/(\lambda\|x\|) = x/\|x\| = \hat x$ — the $\lambda$ cancels, so every positive multiple of $x$ maps to the same normalized point. The fibers of $x \mapsto \hat x$ (restricted to $\mathbb{R}^d\setminus\{0\}$ ) are exactly the open rays from the origin, each ray mapping to a single point on the sphere — this is precisely the projection $\pi$ from Cone Exercise 6, restricted to positive $\lambda$ only (a half-line rather than a full line).

Pro

Prove that $S^n$ is simply connected for $n \geq 2$ (i.e. every loop can be continuously contracted to a point), using a covering argument sketch: why does the argument fail for $n=1$ ?

Solution

Sketch: any continuous loop $\gamma: [0,1] \to S^n$ can, for $n \geq 2$ , be perturbed (via a small homotopy) to miss at least one point $p \in S^n$ — this is possible because a 1-dimensional image (the loop) generically fails to cover an $n$ -dimensional space when $n \geq 2$ (a transversality/dimension-counting argument: a 1-parameter family cannot generically fill an $\geq 2$ -dimensional target). Once the loop misses a point $p$ , $S^n \setminus \{p\}$ is homeomorphic to $\mathbb{R}^n$ via stereographic projection, and $\mathbb{R}^n$ is contractible, so the loop can be contracted within $S^n\setminus\{p\} \subset S^n$ . For $n=1$ , $S^1$ minus a point is homeomorphic to $\mathbb{R}^1$ , which is contractible too — but the argument’s first step fails: a loop on $S^1$ (a 1-dimensional space) generically cannot be perturbed to miss a point while staying a loop that winds around, since a “generic” loop in a 1-dimensional space already covers an open set densely by dimension-counting parity; concretely, the winding-number loops (going around $n\neq0$ times) are not homotopic to a point-missing configuration without changing the winding number itself, which is exactly the obstruction $H_1(S^1) = \mathbb{Z} \neq 0$ records.

Post-LayerNorm activations satisfy $\hat x = (x-\mu)/\sigma$ for per-example mean $\mu$ and std $\sigma$ , then a learned affine rescale $\gamma\hat x + \beta$ . Explain precisely why the rescaled output $\gamma\hat x+\beta$ does not, in general, still lie exactly on a sphere, and under what special case on $\gamma,\beta$ it would.

Solution

$\hat x$ (after mean-subtraction and division by $\sigma$ ) lies on a sphere of fixed radius (depending on the normalization convention), but $\gamma\hat x + \beta$ applies a coordinate-wise scale $\gamma$ and shift $\beta$ . A coordinate-wise (non-uniform) scale by $\gamma$ turns a sphere into an ellipsoid in general (each axis stretched by a different factor $\gamma_i$ ), and adding $\beta \neq 0$ translates it off-center — an ellipsoid centered away from the origin is not a sphere centered at the origin. The output lies exactly on a (possibly different) sphere centered at the origin only in the special case $\beta = 0$ and $\gamma_i = c$ (the same constant) for every coordinate $i$ — i.e. $\gamma$ is a uniform scalar rather than a per-coordinate vector, and there’s no additive shift.

Found in (4 observations · 6 families)

ResNet

ArcFace: Additive Angular Margin Loss for Deep Face Recognition (2019)measured

Additive angular margin gives a constant geodesic margin on the hypersphere

Details

Deng et al. replace SphereFace's multiplicative angular margin with an additive one, cos(theta_yi + m), and give a unified formula covering SphereFace, ArcFace, and CosFace margins as special cases [deng-etal-2019-arcface] Because features are L2-normalized onto a hypersphere of radius s (a geometry imposed by construction), adding directly to the angle yields a constant linear margin with exact correspondence to geodesic distance, unlike SphereFace's nonlinear margin [deng-etal-2019-arcface] A matched ResNet50/CASIA-WebFace ablation shows ArcFace (m=0.5) beating SphereFace and CosFace (LFW 99.53% vs 99.11%/99.51%; AgeDB 95.15% vs 91.70%/94.56%), with m=0.5 optimal [deng-etal-2019-arcface] Scaled to ResNet100, the citable arXiv version reports MegaFace 98.35%/98.48% and IJB-C TAR@FAR=1e-4 of 95.6%; the higher 98.51/98.74 and 97.27 figures come from the later TPAMI/MS1MV3 revision [deng-etal-2019-arcface]

models: ArcFace ResNet50 (trained on CASIA/VGG2), ArcFace ResNet100 (trained on MS1MV3/IBUG-500K) · method: Additive angular-margin softmax loss, Geometric analysis

Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere (2020)measured

Contrastive loss decomposes into alignment plus hypersphere uniformity

Details

Wang & Isola prove that as negative samples grow, the InfoNCE contrastive loss on the unit hypersphere converges to an alignment term (positive pairs close) plus a uniformity term whose minimizer is the uniform distribution on the sphere [wang-isola-2020-alignment-uniformity-hypersphere] The uniformity result is rigorous via classical potential theory: the Gaussian-potential minimizer converges weakly to the true uniform surface measure on S^d [wang-isola-2020-alignment-uniformity-hypersphere] Training directly on weighted alignment plus uniformity with no contrastive loss matches or exceeds standard contrastive pretraining across five model/dataset combinations (full-ImageNet MoCo v2 replacement 67.69% vs 67.5% linear-probe) [wang-isola-2020-alignment-uniformity-hypersphere] The hypersphere geometry is imposed by L2-normalization and the central claim is analytical (about the loss), not a shape discovered in free activations [wang-isola-2020-alignment-uniformity-hypersphere]

models: ResNet-50 (MoCo v2, unsupervised contrastive pretraining, ImageNet) · method: Alignment/Uniformity Loss Decomposition

AlexNet

Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere (2020)measured

Contrastive loss decomposes into alignment plus hypersphere uniformity

Details

models: AlexNet-based encoder (contrastive/alignment-uniformity, STL-10) · method: Alignment/Uniformity Loss Decomposition

Qwen

Shape Happens: Automatic Feature Manifold Discovery in LLMs via Supervised Multi-Dimensional Scaling (2025)measured

Geographic coordinates fit a spherical, not flat, manifold

Details

Tiblias et al. revisit geographic-location decoding and use sMDS to grid-search candidate curvatures (sphere, cylinder, geodesic great-circle embedding) for city latitude/longitude [tiblias-etal-2025] The spherical manifold achieves the lowest stress across all three models tested, with the recovered projection clustering cities by continent in correct relative real-world position [tiblias-etal-2025] This is a competing curvature claim about the same phenomenon Gurnee & Tegmark decoded from a flat plane, kept separate since the two disagree on curvature rather than domain or mechanism [tiblias-etal-2025]

models: Qwen2.5-3B-Instruct · method: Supervised Multi-Dimensional Scaling (SMDS), Geometric analysis

Llama

Shape Happens: Automatic Feature Manifold Discovery in LLMs via Supervised Multi-Dimensional Scaling (2025)measured

Geographic coordinates fit a spherical, not flat, manifold

Details

models: Llama-3.2-3B-Instruct · method: Supervised Multi-Dimensional Scaling (SMDS), Geometric analysis

Gemma

Shape Happens: Automatic Feature Manifold Discovery in LLMs via Supervised Multi-Dimensional Scaling (2025)measured

Geographic coordinates fit a spherical, not flat, manifold

Details

models: Gemma-2-2B-it · method: Supervised Multi-Dimensional Scaling (SMDS), Geometric analysis

SphereFace CNN

SphereFace: Deep Hypersphere Embedding for Face Recognition (2017)measured

A-Softmax forces face embeddings onto a hypersphere with a tunable angular margin

Details

Liu et al. modify softmax into a purely angle-based metric by normalizing classifier weights and removing bias, then multiplicatively compressing the target class's angular region by an integer margin m [liu-etal-2017-sphereface] The learned features construct an angular distance metric equivalent to geodesic distance on a hypersphere, a geometry imposed by the normalization rather than discovered in free activations [liu-etal-2017-sphereface] Sweeping m through 1-4 monotonically improves face verification (LFW 97.88% to 99.42%, YTF 93.1% to 95.0%) and open-set MegaFace identification (Rank-1 54.86% to 72.73%) [liu-etal-2017-sphereface] The 64-layer m=4 model trained only on CASIA-WebFace beats Softmax+Center-Loss and rivals models trained on far larger private datasets [liu-etal-2017-sphereface]

models: SphereFace (64-conv-layer residual CNN, trained on CASIA-WebFace) · method: Angular-margin softmax loss, Geometric analysis