MATH · IN · MODELS

Linear Structures

background definition

Structures in activation space described by linear algebra: directions, subspaces, separating hyperplanes, affine subspaces.

Definition

A linear structure is a subset of Rd\mathbb{R}^d definable by linear-algebraic operations alone: a 1-dimensional direction (a unit vector rfr_f), a kk-dimensional subspace (V=span{e1,,ek}V=\text{span}\{e_1,\ldots,e_k\}), an affine translate of a subspace (x0+Vx_0+V), or a hyperplane ({x:wx=c}\{x:w^\top x=c\}). All are flat (zero curvature) and, except for the affine/hyperplane case with nonzero offset, pass through the origin.

Intuition

Even though a network’s computation is highly non-linear, the set of representations it produces can still be linearly organized — features encoded as directions, classes separated by flat hyperplanes — the same way a non-linear physical process can still produce data that lies, to good approximation, on a straight line or flat plane.

Properties

Linear structures nest by dimension, from simplest to most general:

StructureDimensionPasses through originDefined by
Linear Direction1Yesa single unit vector rfr_f
Linear Subspacek>1k>1Yesan orthonormal basis e1,,eke_1,\ldots,e_k
Affine SubspacekkOnly if x0Vx_0\in Va basis for VV plus an offset x0x_0
Linear Separability (hyperplane)d1d-1Only if b=0b=0a normal vector ww and offset bb
Conceptor (soft ellipsoidal region) (soft/graded)kk (graded)Yesa PSD matrix CC with eigenvalues in [0,1][0,1]
Relation frame (ordered multi-token tuple geometry) (multi-token tuple)rr tokens, per-token dimsN/A (per-instance)an ordered tuple’s centroid, centered shape, spanned subspace, orientation sign

A direction is a 11-dimensional linear subspace; a linear subspace generalizes a direction to k>1k>1 dimensions; an affine subspace generalizes further by dropping the origin-containment requirement; a separating hyperplane is the special case of an affine subspace with codimension exactly 11 (dim=d1\dim = d-1), viewed as a decision boundary rather than a feature-encoding subspace.

Exercises

Base

  1. Is a hyperplane {xRd:wx=c}\{x \in \mathbb{R}^d : w^\top x = c\} (for w0w \neq 0, c0c\neq0) a linear subspace? Is it an affine subspace?
Solution

Not a linear subspace (it does not contain 0\mathbf 0, since w0=0cw^\top 0 = 0 \neq c). It is an affine subspace of dimension d1d-1: it equals x0+Vx_0+V for V=wV=w^\perp (dimension d1d-1 by rank-nullity) and any particular solution x0x_0 with wx0=cw^\top x_0=c.

  1. A single feature direction rfRdr_f \in \mathbb{R}^d with rf=1\|r_f\|=1 is, by itself, a linear subspace of what dimension?
Solution

Dimension 11: span{rf}={λrf:λR}\text{span}\{r_f\} = \{\lambda r_f : \lambda \in \mathbb{R}\} is a 1-dimensional linear subspace, and a “direction” is simply this subspace together with a choice of unit-norm generator (the sign/normalization convention, not an intrinsic part of the subspace itself).

Middle

  1. Prove that every kk-dimensional linear subspace VRdV \subset \mathbb{R}^d can be written as the intersection of dkd-k hyperplanes through the origin.
Solution

Since dimV=k\dim V = k, its orthogonal complement VV^\perp has dimension dkd-k. Pick an orthonormal basis w1,,wdkw_1,\ldots,w_{d-k} of VV^\perp. Then V=i=1dk{x:wix=0}V = \bigcap_{i=1}^{d-k} \{x : w_i^\top x = 0\}: any xVx\in V satisfies wix=0w_i^\top x=0 for all ii (since wiVw_i \perp V), and conversely any xx orthogonal to all of VV^\perp‘s basis vectors is orthogonal to all of VV^\perp, hence lies in (V)=V(V^\perp)^\perp = V. Each {x:wix=0}\{x:w_i^\top x=0\} is a hyperplane through the origin, so VV is the intersection of dkd-k such hyperplanes.

  1. Show that the set of directions {rRd:r=1}\{r \in \mathbb{R}^d : \|r\|=1\} is not itself a linear structure (not a subspace, not an affine subspace) even though each individual rr spans one.
Solution

The unit sphere Sd1={r:r=1}S^{d-1} = \{r:\|r\|=1\} is not closed under addition or scalar multiplication (e.g. r+r=2rr+r = 2r has norm 212\neq1, so r+rSd1r+r\notin S^{d-1} even though rSd1r\in S^{d-1}), so it fails to be a linear subspace. It is also not an affine subspace: affine subspaces are unbounded (whenever dim1\dim\geq1) or single points, but Sd1S^{d-1} is bounded and, for d2d\geq2, is not a single point — a curved manifold (Sphere), not a flat linear structure. Each individual rr spans a 1D linear subspace span{r}\text{span}\{r\}, but the collection of all such rr (the sphere itself) is a fundamentally different, curved object.

Pro

  1. Let V1,V2RdV_1,V_2 \subset \mathbb{R}^d be linear subspaces with dimV1=k1\dim V_1=k_1, dimV2=k2\dim V_2=k_2. Give the tightest general bounds on dim(V1V2)\dim(V_1\cap V_2) in terms of k1,k2,dk_1,k_2,d, and justify.
Solution

max(0,k1+k2d)    dim(V1V2)    min(k1,k2)\max(0,\, k_1+k_2-d) \;\leq\; \dim(V_1\cap V_2) \;\leq\; \min(k_1,k_2). Upper bound: V1V2V1V_1\cap V_2 \subseteq V_1 and V2\subseteq V_2, so its dimension is at most the smaller of the two. Lower bound: from the dimension formula dim(V1+V2)=dimV1+dimV2dim(V1V2)\dim(V_1+V_2) = \dim V_1+\dim V_2-\dim(V_1\cap V_2), and since V1+V2RdV_1+V_2\subseteq\mathbb{R}^d forces dim(V1+V2)d\dim(V_1+V_2)\leq d, rearranging gives dim(V1V2)=k1+k2dim(V1+V2)k1+k2d\dim(V_1\cap V_2) = k_1+k_2-\dim(V_1+V_2) \geq k_1+k_2-d; combined with dim(V1V2)0\dim(V_1\cap V_2)\geq 0 always, this gives the stated lower bound. Both bounds are achieved: e.g. two generic subspaces achieve the lower bound, and V1V2V_1\subseteq V_2 achieves the upper bound with equality.

  1. A “concept erasure” operation replaces xx with PVx=xPVxP_{V^\perp}x = x - P_Vx for a kk-dimensional subspace VV believed to encode a concept. Prove that this operation is idempotent (PV(PVx)=PVxP_{V^\perp}(P_{V^\perp}x) = P_{V^\perp}x) and that it commutes with re-erasing a subspace VVV' \subseteq V: P(V)(PVx)=PVxP_{(V')^\perp}(P_{V^\perp}x) = P_{V^\perp}x whenever VVV'\subseteq V.
Solution

Idempotence: PVP_{V^\perp} is an orthogonal projection onto VV^\perp; for any orthogonal projection PP, P(Px)=PxP(Px) = Px since PxPx \in range(P)(P) already, and a projection fixes its own range pointwise (a defining property of orthogonal — indeed any idempotent — projections, P2=PP^2=P). Second claim: since VVV'\subseteq V, we have V(V)V^\perp \subseteq (V')^\perp (larger subspace has smaller, more restrictive orthogonal complement… precisely: taking orthogonal complements reverses inclusions). PVxV(V)P_{V^\perp}x \in V^\perp \subseteq (V')^\perp, and since (V)(V')^\perp‘s projection fixes every vector already inside (V)(V')^\perp (same idempotence-type fact, applied to a vector already in the target subspace rather than to a repeated application of the same projection), P(V)(PVx)=PVxP_{(V')^\perp}(P_{V^\perp}x) = P_{V^\perp}x.

Found in (0 observations · 0 families)

No observations confirm this structure yet.