Affine Subspace

structures / Manifolds / Affine Subspace

OBJECTgeometric-objectsubsetaffineK:zeromanifoldbeginnerhow it's classified →

Linear subspace with an offset: A = x₀ + V. Flat, open manifold of zero curvature. Unlike a linear subspace, need not pass through the origin.

Replicationcomputed from the corpus — never hand-assigned

6 papers · no shared authors5 architecture classes · across papers2 domains · across papers10 model families · across papers

Filled = two or more values reported by papers that share no author — replication. Outlined = two or more values, but all from a single study — breadth, not replication. Grey = a single value. Derived from paper authorship and each model's architecture class, domain and family; it updates itself when a paper is added.

Definition

An affine subspace $A = x_0 + V = \{x_0 + v : v \in V\}$ , where $V \subseteq \mathbb{R}^d$ is a linear subspace and $x_0 \in \mathbb{R}^d$ is a fixed offset. $\dim A := \dim V$ .

Intuition

A line through the origin is a linear subspace; slide it sideways so it no longer touches the origin, and it becomes an affine subspace of the same dimension. The offset $x_0$ carries no special status — any point of $A$ could serve as the “basepoint,” since $A = x_1 + V$ for any $x_1 \in A$ .

Properties

Contains the origin iff $x_0 \in V$ . A linear subspace always contains $\mathbf{0}$ ; an affine subspace contains $\mathbf{0}$ only in the special case $x_0 \in V$ (equivalently, $A=V$ ).
Not closed under addition in general. If $a_1,a_2 \in A$ then $a_1+a_2 = 2x_0 + (v_1+v_2) \notin A$ unless $x_0 \in V$ — an affine subspace is closed under affine combinations ( $\sum_i c_i a_i$ with $\sum_i c_i = 1$ ), not arbitrary linear combinations.
Flat, zero curvature, open (non-compact) unless $V=\{0\}$ . As a translate of a linear subspace, $A$ inherits $V$ ‘s flat Euclidean geometry exactly.
Basepoint-independence. For any $x_1 \in A$ , $A = x_1 + V$ — the decomposition $A = x_0+V$ is not unique in $x_0$ (any point of $A$ works), only $V$ (the direction of $A$ ) is an invariant of $A$ itself.
Parallel affine subspaces. Two affine subspaces $x_0+V$ and $x_1+V$ with the same $V$ but different offsets are parallel and disjoint whenever $x_1 - x_0 \notin V$ ; they partition $\mathbb{R}^d$ into disjoint parallel copies of $V$ when $V$ has codimension $\geq 1$ .
Correspondence with biased linear probes. A linear probe with bias, $\{x : w^\top x + b = 0\}$ , is exactly an affine hyperplane: the set $\{x : w^\top x = -b\}$ , which is $x_0 + V$ for $V = w^\perp$ (the linear hyperplane through the origin orthogonal to $w$ ) and any particular solution $x_0$ of $w^\top x_0 = -b$ .

Caveat: a latent space can look locally affine at every point while still being globally warped

Choi, Lee, Yoon, Park, Hwang & Kang (2021) study StyleGAN2’s intermediate W latent space, where prior work (GANSpace, SeFa) implicitly treats a single globally-fit direction set as valid everywhere — the working assumption behind fitting one affine chart to the whole space. Computing a genuinely local tangent-direction basis at each point (Jacobian-SVD of the generator’s partial map, “Local Basis”) and comparing these local bases across different points via a Grassmannian manifold distance, they find nearby points’ local bases agree closely but distant points’ diverge significantly more than chance — i.e. the space’s local linear structure is real at each point, but does not compose into one consistent global affine subspace. This is why global methods degrade (measured by rising FID and visible “image collapse”) under large perturbations while following the local basis instead does not: the traversal direction that is locally optimal changes as you move, exactly as it would on a genuinely curved manifold rather than a flat affine one. See stylegan2-w-space-is-globally-warped-local- jacobian-subspaces-diverge-with-grassmannian-distance-and-following- them-avoids-traversal-collapse.

Exercises

Base

Is $A = \{(1,t) : t \in \mathbb{R}\} \subset \mathbb{R}^2$ a linear subspace? Is it an affine subspace? If so, identify $x_0$ and $V$ .

Solution

Not a linear subspace (it doesn’t contain $(0,0)$ , and e.g. $(1,0)+(1,1)=(2,1)\notin A$ ). It is an affine subspace: $A = (1,0) + \{(0,t):t\in\mathbb{R}\}$ , so $x_0=(1,0)$ and $V = \{(0,t):t\in\mathbb{R}\} = \text{span}\{(0,1)\}$ , a 1-dimensional linear subspace.

For the affine subspace $A$ from Exercise 1, is $(1,5)$ a valid alternative choice of basepoint — i.e. does $A = (1,5)+V$ for the same $V$ ?

Solution

Yes. $(1,5) \in A$ (take $t=5$ ), and $(1,5)+V = \{(1,5+t):t\in\mathbb{R}\} = \{(1,s):s\in\mathbb{R}\} = A$ , matching the original set exactly — any point of $A$ is a valid basepoint.

Middle

Prove that an affine subspace $A=x_0+V$ is closed under affine combinations: if $a_1,\ldots,a_k \in A$ and $c_1,\ldots,c_k \in \mathbb{R}$ with $\sum_i c_i = 1$ , then $\sum_i c_i a_i \in A$ .

Solution

Write $a_i = x_0+v_i$ for $v_i \in V$ . Then $\sum_i c_i a_i = \sum_i c_i(x_0+v_i) = \big(\sum_i c_i\big) x_0 + \sum_i c_i v_i = x_0 + \sum_i c_i v_i$ (using $\sum_i c_i=1$ ). Since $V$ is a linear subspace and each $v_i \in V$ , $\sum_i c_i v_i \in V$ (closed under arbitrary linear combinations, including negative $c_i$ ). So $\sum_i c_i a_i = x_0 + (\text{something in } V) \in A$ .

Two affine subspaces $A_1 = x_0+V$ and $A_2=x_1+V$ share the same direction $V$ but $x_1-x_0 \notin V$ . Prove $A_1 \cap A_2 = \emptyset$ .

Solution

Suppose $z \in A_1\cap A_2$ . Then $z = x_0+v_0 = x_1+v_1$ for some $v_0,v_1\in V$ , so $x_1-x_0 = v_0-v_1 \in V$ (since $V$ is closed under subtraction). This contradicts the assumption $x_1-x_0\notin V$ . Hence no such $z$ exists, i.e. $A_1\cap A_2=\emptyset$ .

Pro

Show that a linear probe’s decision boundary $\{x : w^\top x+b=0\}$ (for $w\neq 0$ ) is an affine subspace of dimension $d-1$ (a hyperplane), by exhibiting explicit $x_0$ and $V$ .

Solution

Take $V = \{v : w^\top v = 0\} = w^\perp$ , the linear hyperplane orthogonal to $w$ (dimension $d-1$ , since it’s the kernel of the nonzero linear functional $w^\top(\cdot)$ , rank-nullity gives $\dim V = d - 1$ ). Take any particular solution $x_0$ with $w^\top x_0 = -b$ — e.g. $x_0 = -\frac{b}{\|w\|^2} w$ (check: $w^\top x_0 = -\frac{b}{\|w\|^2}\|w\|^2 = -b$ ✓.) Then for any $x$ with $w^\top x = -b$ : $w^\top(x-x_0) = -b-(-b) = 0$ , so $x - x_0 \in V$ , i.e. $x = x_0+v$ for some $v\in V$ . Conversely any $x_0+v$ (with $v\in V$ ) satisfies $w^\top(x_0+v) = -b+0=-b$ . So $\{x:w^\top x=-b\} = x_0+V$ , an affine subspace of dimension $d-1$ .

Let $A = x_0+V$ with $\dim V = k$ . Prove that the set of valid basepoints for $A$ (i.e. $\{x_1 \in \mathbb{R}^d : A = x_1+V\}$ ) is exactly $A$ itself — not a larger or smaller set.

Solution

( $\subseteq$ ) If $A = x_1+V$ , then in particular $x_1 = x_1+0 \in x_1+V = A$ (since $0\in V$ ), so $x_1\in A$ . ( $\supseteq$ ) Conversely, suppose $x_1 \in A = x_0+V$ , so $x_1 = x_0+v_1$ for some $v_1\in V$ . Then $x_1+V = x_0+v_1+V = x_0+V$ (since $v_1+V=V$ : adding a fixed element of a linear subspace to the whole subspace just permutes it back to itself, as $V$ is closed under addition and $-v_1\in V$ too) $=A$ . So every $x_1\in A$ is a valid basepoint, and by the first direction, only points of $A$ are. Hence the set of valid basepoints is exactly $A$ .

Found in (6 observations · 8 families)

Llama

Do Sparse Autoencoders Capture Concept Manifolds? (2026)measured

Open-ended scalar concepts trace affine 1D lines

Details

Bhalla et al. find simple open-ended scalar concepts (temperature, age) trace flat open 1D lines in Llama-3.1-8B's layer-19 residual stream, via PCA across sampled prompt values [bhalla-etal-2026] Causal steering along interpolated points between binned centroids is validated only for temperature, color, and day-of-week (e.g. freezing to cold to warm to hot), producing smooth monotonic output shifts [bhalla-etal-2026] Age and years show the same open 1D structure by PCA but were not steered, so their causal load-bearing status is not established here [bhalla-etal-2026] Formality, sentence length, and a synthetic political-bias score show similar roughly-1D structure with PCA evidence only [bhalla-etal-2026]

models: Llama-3.1-8B · method: PCA, Activation Steering (Addition)

Language Models Represent Space and Time (2024), Symmetry in Language Statistics Shapes the Geometry of Model Representations (2026)measured

Geographic coordinates are represented on a flat affine plane

Details

Latitude and longitude are linearly decodable from a flat 2D affine plane in the residual stream, unlike the cyclic circle used for calendar time in the same models [gurnee-tegmark-2024] Karkada et al. independently derive the same flat-plane geometry analytically as the D=2 open-boundary case of translation-symmetry theory, confirmed in word embeddings and Gemma 2 2B for US cities [karkada-etal-2026] A later analysis fits a spherical rather than flat geometry to the same data, kept as a separate observation since the two disagree on curvature [karkada-etal-2026]

models: Llama-2-7B, Llama-2-13B, Llama-2-70B · method: Linear probing, PCA, Spectral analysis of the co-occurrence/PMI matrix, Analytical derivation

Refusal in LLMs is an Affine Function (2024)measured

Refusal is better described as an affine function than a direction

Details

Marshall et al. generalize the linear direction-ablation account of refusal to an affine one, treating the projection coefficient as an affine function with a nonzero offset derived from the harmless-class mean rather than the origin [marshall-etal-2024] They argue purely linear methods give no guarantee the resulting vector lands in a meaningful location [marshall-etal-2024] Directional ablation alone breaks a non-transformer RWKV (Eagle) model into incoherent output, while their Affine Concept Editing (ACE), combining ablation with offset-based activation addition, fixes it [marshall-etal-2024] This contests the single-direction account and offers an alternative to the concept-cone account of the same phenomenon [marshall-etal-2024]

models: Llama-3-8B-Instruct, Llama-3-70B-Instruct · method: Difference-in-means direction, Causal interventions (steering)

Do Sparse Autoencoders Capture Concept Manifolds? (2026)measured

SAEs dilute continuous manifolds rather than compactly capturing them

Details

Bhalla et al. formalize manifold capture (a small fixed group of decoder atoms spanning the manifold, consistently reselected by the encoder) and identify three regimes: compact capture, tiling/shattering, and intermediate dilution [bhalla-etal-2026] Training five SAE architectures on Llama-3.1-8B layer-19 activations, variance explained by a restricted atom group plateaus well beyond each manifold's ambient dimension, so none achieve compact capture [bhalla-etal-2026] Features behave like overlapping population-code tuning curves that redundantly tile the manifold, placing all tested SAEs in the dilution regime [bhalla-etal-2026] Ising-coactivation analysis on SAE codes recovers known manifolds (temperature, colors, political bias) unsupervised and surfaces a new epistemic-uncertainty manifold [bhalla-etal-2026]

models: Llama-3.1-8B · method: Sparse Autoencoders (SAE), Manifold capture theory (subspace capture / tiling / dilution), Ising coactivation analysis, PCA

Pythia

Language Models Represent Space and Time (2024), Symmetry in Language Statistics Shapes the Geometry of Model Representations (2026)measured

Geographic coordinates are represented on a flat affine plane

Details

models: Pythia-6.9B · method: Linear probing, PCA, Spectral analysis of the co-occurrence/PMI matrix, Analytical derivation

Gemma

Language Models Represent Space and Time (2024), Symmetry in Language Statistics Shapes the Geometry of Model Representations (2026)measured

Geographic coordinates are represented on a flat affine plane

Details

models: Gemma-2-2B, EmbeddingGemma · method: Linear probing, PCA, Spectral analysis of the co-occurrence/PMI matrix, Analytical derivation

word2vec

Language Models Represent Space and Time (2024), Symmetry in Language Statistics Shapes the Geometry of Model Representations (2026)measured

Geographic coordinates are represented on a flat affine plane

Details

models: word2vec (trained on Wikipedia) · method: Linear probing, PCA, Spectral analysis of the co-occurrence/PMI matrix, Analytical derivation

DeBERTa

More than Correlation: Do Large Language Models Learn Causal Representations of Space? (2023)measured

Geographic representations are causally used for prediction, not just decodable

Details

Chen et al. extend the linear-geography finding to DeBERTa-v2-xxlarge and GPT-Neo-1.3B, where non-linear probes substantially outperform linear ones (unlike Gurnee & Tegmark) and a Haversine-based GeoDist loss yields probes whose predicted maps better resemble true geography [chen-etal-2023] RSA on country-level activation distances correlates modestly with real distance (Kendall tau 0.20-0.21), flagged by the authors as suggestive not conclusive [chen-etal-2023] Probe-gradient perturbation establishes causality: sharpening a city's alignment with its true coordinates improves country classification while gradient ascent degrades it, and cross-city perturbation shifts logits toward the target country [chen-etal-2023] Applied to GPT-Neo's own next-token prediction, gradient ascent causes a large drop in country-token accuracy and descent a small significant improvement, so the language-modeling objective itself depends on the spatial representation [chen-etal-2023] Scope: the models are smaller and older (both under 2B), which may limit how strongly the causal finding generalizes to frontier scale [chen-etal-2023]

models: DeBERTa-v2-xxlarge · method: Representational Similarity Analysis (RSA), Linear probing, Mlp probing, Probe-Gradient Perturbation

GPT-Neo

More than Correlation: Do Large Language Models Learn Causal Representations of Space? (2023)measured

Geographic representations are causally used for prediction, not just decodable

Details

models: GPT-Neo-1.3B · method: Representational Similarity Analysis (RSA), Linear probing, Mlp probing, Probe-Gradient Perturbation

RWKV

Refusal in LLMs is an Affine Function (2024)measured

Refusal is better described as an affine function than a direction

Details

models: Hermes Eagle RWKV v5 (7B) · method: Difference-in-means direction, Causal interventions (steering)

StyleGAN

Do Not Escape From the Manifold: Discovering the Local Coordinates on the Latent Space of GANs (2021)measured

StyleGAN2's W space is globally warped; local bases avoid traversal collapse

Details

Choi et al. compute a Local Basis at each W-space point via SVD of the noise-to-W Jacobian, giving local principal-variation directions rather than one global set (as in GANSpace or SeFa) [choi-etal-2021-local-basis-gan-manifold] Grassmannian distance between top-k Local Basis subspaces shows nearby points are far more similar than random points, themselves distinct from a random-subspace null, so W space is globally warped rather than flat/affine [choi-etal-2021-local-basis-gan-manifold] Traversing along the Local Basis keeps FID low under strong perturbation while global methods' FID rises sharply (image collapse), a measured causal effect of respecting local geometry [choi-etal-2021-local-basis-gan-manifold]

models: StyleGAN2 (trained on FFHQ, 1024x1024) · method: Jacobian spectral direction analysis, Subspace-angle alignment, Causal interventions (steering)