Definition
A linear structure is a subset of definable by linear-algebraic operations alone: a 1-dimensional direction (a unit vector ), a -dimensional subspace (), an affine translate of a subspace (), or a hyperplane (). All are flat (zero curvature) and, except for the affine/hyperplane case with nonzero offset, pass through the origin.
Intuition
Even though a network’s computation is highly non-linear, the set of representations it produces can still be linearly organized — features encoded as directions, classes separated by flat hyperplanes — the same way a non-linear physical process can still produce data that lies, to good approximation, on a straight line or flat plane.
Properties
Linear structures nest by dimension, from simplest to most general:
| Structure | Dimension | Passes through origin | Defined by |
|---|---|---|---|
| Linear Direction | 1 | Yes | a single unit vector |
| Linear Subspace | Yes | an orthonormal basis | |
| Affine Subspace | Only if | a basis for plus an offset | |
| Linear Separability (hyperplane) | Only if | a normal vector and offset | |
| Conceptor (soft ellipsoidal region) (soft/graded) | (graded) | Yes | a PSD matrix with eigenvalues in |
| Relation frame (ordered multi-token tuple geometry) (multi-token tuple) | tokens, per-token dims | N/A (per-instance) | an ordered tuple’s centroid, centered shape, spanned subspace, orientation sign |
A direction is a -dimensional linear subspace; a linear subspace generalizes a direction to dimensions; an affine subspace generalizes further by dropping the origin-containment requirement; a separating hyperplane is the special case of an affine subspace with codimension exactly (), viewed as a decision boundary rather than a feature-encoding subspace.
Exercises
Base
- Is a hyperplane (for , ) a linear subspace? Is it an affine subspace?
Solution
Not a linear subspace (it does not contain , since ). It is an affine subspace of dimension : it equals for (dimension by rank-nullity) and any particular solution with .
- A single feature direction with is, by itself, a linear subspace of what dimension?
Solution
Dimension : is a 1-dimensional linear subspace, and a “direction” is simply this subspace together with a choice of unit-norm generator (the sign/normalization convention, not an intrinsic part of the subspace itself).
Middle
- Prove that every -dimensional linear subspace can be written as the intersection of hyperplanes through the origin.
Solution
Since , its orthogonal complement has dimension . Pick an orthonormal basis of . Then : any satisfies for all (since ), and conversely any orthogonal to all of ‘s basis vectors is orthogonal to all of , hence lies in . Each is a hyperplane through the origin, so is the intersection of such hyperplanes.
- Show that the set of directions is not itself a linear structure (not a subspace, not an affine subspace) even though each individual spans one.
Solution
The unit sphere is not closed under addition or scalar multiplication (e.g. has norm , so even though ), so it fails to be a linear subspace. It is also not an affine subspace: affine subspaces are unbounded (whenever ) or single points, but is bounded and, for , is not a single point — a curved manifold (Sphere), not a flat linear structure. Each individual spans a 1D linear subspace , but the collection of all such (the sphere itself) is a fundamentally different, curved object.
Pro
- Let be linear subspaces with , . Give the tightest general bounds on in terms of , and justify.
Solution
. Upper bound: and , so its dimension is at most the smaller of the two. Lower bound: from the dimension formula , and since forces , rearranging gives ; combined with always, this gives the stated lower bound. Both bounds are achieved: e.g. two generic subspaces achieve the lower bound, and achieves the upper bound with equality.
- A “concept erasure” operation replaces with for a -dimensional subspace believed to encode a concept. Prove that this operation is idempotent () and that it commutes with re-erasing a subspace : whenever .
Solution
Idempotence: is an orthogonal projection onto ; for any orthogonal projection , since range already, and a projection fixes its own range pointwise (a defining property of orthogonal — indeed any idempotent — projections, ). Second claim: since , we have (larger subspace has smaller, more restrictive orthogonal complement… precisely: taking orthogonal complements reverses inclusions). , and since ‘s projection fixes every vector already inside (same idempotence-type fact, applied to a vector already in the target subspace rather than to a repeated application of the same projection), .