MATH · IN · MODELS

Linear Structures

background definition

Structures in activation space described by linear algebra: directions, subspaces, separating hyperplanes, affine subspaces.

Definition

A linear structure is a subset of Rd\mathbb{R}^d definable by linear-algebraic operations alone: a 1-dimensional direction (a unit vector rfr_f), a kk-dimensional subspace (V=span{e1,…,ek}V=\text{span}\{e_1,\ldots,e_k\}), an affine translate of a subspace (x0+Vx_0+V), or a hyperplane ({x:w⊤x=c}\{x:w^\top x=c\}). All are flat (zero curvature) and, except for the affine/hyperplane case with nonzero offset, pass through the origin.

Intuition

Even though a network’s computation is highly non-linear, the set of representations it produces can still be linearly organized — features encoded as directions, classes separated by flat hyperplanes — the same way a non-linear physical process can still produce data that lies, to good approximation, on a straight line or flat plane.

Properties

Linear structures nest by dimension, from simplest to most general:

StructureDimensionPasses through originDefined by
Linear Direction1Yesa single unit vector rfr_f
Linear Subspacek>1k>1Yesan orthonormal basis e1,…,eke_1,\ldots,e_k
Affine SubspacekkOnly if x0∈Vx_0\in Va basis for VV plus an offset x0x_0
Linear Separability (hyperplane)d−1d-1Only if b=0b=0a normal vector ww and offset bb
Conceptor (soft ellipsoidal region) (soft/graded)kk (graded)Yesa PSD matrix CC with eigenvalues in [0,1][0,1]
Relation frame (ordered multi-token tuple geometry) (multi-token tuple)rr tokens, per-token dimsN/A (per-instance)an ordered tuple’s centroid, centered shape, spanned subspace, orientation sign

A direction is a 11-dimensional linear subspace; a linear subspace generalizes a direction to k>1k>1 dimensions; an affine subspace generalizes further by dropping the origin-containment requirement; a separating hyperplane is the special case of an affine subspace with codimension exactly 11 (dim⁡=d−1\dim = d-1), viewed as a decision boundary rather than a feature-encoding subspace.

Exercises

Base

  1. Is a hyperplane {x∈Rd:w⊤x=c}\{x \in \mathbb{R}^d : w^\top x = c\} (for w≠0w \neq 0, c≠0c\neq0) a linear subspace? Is it an affine subspace?
Solution

Not a linear subspace (it does not contain 0\mathbf 0, since w⊤0=0≠cw^\top 0 = 0 \neq c). It is an affine subspace of dimension d−1d-1: it equals x0+Vx_0+V for V=w⊥V=w^\perp (dimension d−1d-1 by rank-nullity) and any particular solution x0x_0 with w⊤x0=cw^\top x_0=c.

  1. A single feature direction rf∈Rdr_f \in \mathbb{R}^d with ∥rf∥=1\|r_f\|=1 is, by itself, a linear subspace of what dimension?
Solution

Dimension 11: span{rf}={λrf:λ∈R}\text{span}\{r_f\} = \{\lambda r_f : \lambda \in \mathbb{R}\} is a 1-dimensional linear subspace, and a “direction” is simply this subspace together with a choice of unit-norm generator (the sign/normalization convention, not an intrinsic part of the subspace itself).

Middle

  1. Prove that every kk-dimensional linear subspace V⊂RdV \subset \mathbb{R}^d can be written as the intersection of d−kd-k hyperplanes through the origin.
Solution

Since dim⁡V=k\dim V = k, its orthogonal complement V⊥V^\perp has dimension d−kd-k. Pick an orthonormal basis w1,…,wd−kw_1,\ldots,w_{d-k} of V⊥V^\perp. Then V=⋂i=1d−k{x:wi⊤x=0}V = \bigcap_{i=1}^{d-k} \{x : w_i^\top x = 0\}: any x∈Vx\in V satisfies wi⊤x=0w_i^\top x=0 for all ii (since wi⊥Vw_i \perp V), and conversely any xx orthogonal to all of V⊥V^\perp‘s basis vectors is orthogonal to all of V⊥V^\perp, hence lies in (V⊥)⊥=V(V^\perp)^\perp = V. Each {x:wi⊤x=0}\{x:w_i^\top x=0\} is a hyperplane through the origin, so VV is the intersection of d−kd-k such hyperplanes.

  1. Show that the set of directions {r∈Rd:∥r∥=1}\{r \in \mathbb{R}^d : \|r\|=1\} is not itself a linear structure (not a subspace, not an affine subspace) even though each individual rr spans one.
Solution

The unit sphere Sd−1={r:∥r∥=1}S^{d-1} = \{r:\|r\|=1\} is not closed under addition or scalar multiplication (e.g. r+r=2rr+r = 2r has norm 2≠12\neq1, so r+r∉Sd−1r+r\notin S^{d-1} even though r∈Sd−1r\in S^{d-1}), so it fails to be a linear subspace. It is also not an affine subspace: affine subspaces are unbounded (whenever dim⁡≥1\dim\geq1) or single points, but Sd−1S^{d-1} is bounded and, for d≥2d\geq2, is not a single point — a curved manifold (Sphere), not a flat linear structure. Each individual rr spans a 1D linear subspace span{r}\text{span}\{r\}, but the collection of all such rr (the sphere itself) is a fundamentally different, curved object.

Pro

  1. Let V1,V2⊂RdV_1,V_2 \subset \mathbb{R}^d be linear subspaces with dim⁡V1=k1\dim V_1=k_1, dim⁡V2=k2\dim V_2=k_2. Give the tightest general bounds on dim⁡(V1∩V2)\dim(V_1\cap V_2) in terms of k1,k2,dk_1,k_2,d, and justify.
Solution

max⁡(0, k1+k2−d)  ≤  dim⁡(V1∩V2)  ≤  min⁡(k1,k2)\max(0,\, k_1+k_2-d) \;\leq\; \dim(V_1\cap V_2) \;\leq\; \min(k_1,k_2). Upper bound: V1∩V2⊆V1V_1\cap V_2 \subseteq V_1 and ⊆V2\subseteq V_2, so its dimension is at most the smaller of the two. Lower bound: from the dimension formula dim⁡(V1+V2)=dim⁡V1+dim⁡V2−dim⁡(V1∩V2)\dim(V_1+V_2) = \dim V_1+\dim V_2-\dim(V_1\cap V_2), and since V1+V2⊆RdV_1+V_2\subseteq\mathbb{R}^d forces dim⁡(V1+V2)≤d\dim(V_1+V_2)\leq d, rearranging gives dim⁡(V1∩V2)=k1+k2−dim⁡(V1+V2)≥k1+k2−d\dim(V_1\cap V_2) = k_1+k_2-\dim(V_1+V_2) \geq k_1+k_2-d; combined with dim⁡(V1∩V2)≥0\dim(V_1\cap V_2)\geq 0 always, this gives the stated lower bound. Both bounds are achieved: e.g. two generic subspaces achieve the lower bound, and V1⊆V2V_1\subseteq V_2 achieves the upper bound with equality.

  1. A “concept erasure” operation replaces xx with PV⊥x=x−PVxP_{V^\perp}x = x - P_Vx for a kk-dimensional subspace VV believed to encode a concept. Prove that this operation is idempotent (PV⊥(PV⊥x)=PV⊥xP_{V^\perp}(P_{V^\perp}x) = P_{V^\perp}x) and that it commutes with re-erasing a subspace V′⊆VV' \subseteq V: P(V′)⊥(PV⊥x)=PV⊥xP_{(V')^\perp}(P_{V^\perp}x) = P_{V^\perp}x whenever V′⊆VV'\subseteq V.
Solution

Idempotence: PV⊥P_{V^\perp} is an orthogonal projection onto V⊥V^\perp; for any orthogonal projection PP, P(Px)=PxP(Px) = Px since Px∈Px \in range(P)(P) already, and a projection fixes its own range pointwise (a defining property of orthogonal — indeed any idempotent — projections, P2=PP^2=P). Second claim: since V′⊆VV'\subseteq V, we have V⊥⊆(V′)⊥V^\perp \subseteq (V')^\perp (larger subspace has smaller, more restrictive orthogonal complement… precisely: taking orthogonal complements reverses inclusions). PV⊥x∈V⊥⊆(V′)⊥P_{V^\perp}x \in V^\perp \subseteq (V')^\perp, and since (V′)⊥(V')^\perp‘s projection fixes every vector already inside (V′)⊥(V')^\perp (same idempotence-type fact, applied to a vector already in the target subspace rather than to a repeated application of the same projection), P(V′)⊥(PV⊥x)=PV⊥xP_{(V')^\perp}(P_{V^\perp}x) = P_{V^\perp}x.

Found in (0 observations · 0 families)

No observations confirm this structure yet.