Statement

For each human-interpretable feature $f$, there exists a unit vector $r_f\in\mathbb{R}^d$, $\|r_f\|=1$, such that the projection $r_f\cdot x$ monotonically reflects the “value” of $f$ in the context generating activation $x$ (see Linear Direction).

Strong (operational) form: if $f_1,f_2$ are independent features with directions $r_1,r_2$, a system can “edit” the value of $f_1$ without changing $f_2$ via a linear intervention in activation space, and semantic operations correspond to linear operations — antonymy to reflection through a hyperplane perpendicular to $r_f$, negation to a sign flip of the projection.

Intuition

The claim is not that the underlying computation is linear — it plainly is not, for any sufficiently expressive non-linear system. The claim is that the geometry of the representation space itself is linearly organized: concepts occupy directions, and combining concepts corresponds to adding vectors, even though the process that produced those vectors was highly non-linear.

Properties

• A geometric claim, decoupled from the computation that produces it. LRH concerns the structure of the set of representations, not the function mapping inputs to representations.
• Implies linear separability, but is strictly stronger. If $r_f$ exists as stated, then any two classes differing in the value of $f$ are linearly separable by a hyperplane orthogonal to $r_f$ (see Linear Separability) — but separability alone (existence of some hyperplane) does not imply the stronger operational claim that the same $r_f$ continues to track graded feature value, or that arithmetic over $r_f$ is meaningful.
• Predicts a specific parallelogram structure for analogies. If “gender” is a direction $r_g$ and “royalty” a direction $r_r$, with $\text{man},\text{woman},\text{king},\text{queen}$ positioned as $\text{man}+r_g\approx\text{woman}$ and $\text{man}+r_r\approx\text{king}$, the strong form predicts $\text{king}-\text{man}+\text{woman}\approx\text{queen}$ — a parallelogram in activation space, testable independently of whichever specific words are chosen.
• In tension with superposition. If a $d$-dimensional space must encode $\gg d$ features, the pigeonhole principle rules out $\gg d$ mutually orthogonal directions, forcing either shared (non-orthogonal, interfering) directions per feature or a breakdown of the “one clean direction per feature” picture — LRH’s strong form is easiest to maintain exactly for features that are comparatively rare or highly salient relative to the ambient dimension.
• Linearity can be an artifact of the experimenter’s choice of target labelling, not just the network’s computation. Whether a feature “is” linear or non-linear can depend on which semantically-equivalent relabelling of the classification target a researcher happens to probe for — Nanda, Lee & Wattenberg (2023) show OthelloGPT’s board state, previously reported as non-linearly encoded when probed as absolute colour (BLACK/WHITE/EMPTY), becomes near-perfectly linearly decodable once relabelled relative to the current player (MINE/YOURS/EMPTY) — the same underlying computation, a different human-chosen frame. A negative linearity result should therefore be treated as evidence about the probed labelling, not necessarily about the network, until alternative natural framings have been ruled out.
• Underdetermined without a causal test. A direction extracted by fitting a classifier to correlate with a feature is a candidate $r_f$; confirming it satisfies the operational form of LRH (edits along $r_f$ change only $f$, predictably) requires an intervention, not just a fit — see Linear Direction‘s independence discussion.

Exercises

Base

1. If $\text{man} + r_g = \text{woman}$ and $\text{king}+r_g=\text{queen}$ exactly (same $r_g$ for both pairs), express $\text{queen}-\text{king}$ in terms of $\text{woman}-\text{man}$.

2. Using the same exact relations, derive $\text{king}-\text{man}+\text{woman}=\text{queen}$.

Middle

3. Suppose $r_1,r_2$ are two feature directions with $r_1\cdot r_2 = 0.3$ (not orthogonal). An intervention adds $\alpha r_1$ to $x$. By how much does the projection $r_2\cdot x$ change, and what does this imply about “editing $f_1$ without changing $f_2$” when the directions are not orthogonal?

4. Superposition forces $N \gg d$ features into $d$ dimensions. Using a bound on the maximum number of pairwise-near-orthogonal unit vectors in $\mathbb{R}^d$ (that a set of unit vectors with pairwise dot product $\leq \epsilon$ in absolute value has size at most exponential in $d$, for fixed small $\epsilon>0$, by a packing/Johnson–Lindenstrauss-type argument), explain qualitatively why LRH’s strong form (fully orthogonal, non-interfering directions per feature) becomes easier, not harder, to maintain as $\epsilon$ tolerance is relaxed from “exactly $0$” to “small but nonzero.”

Pro

5. Formalize ”$f_1,f_2$ can be independently edited” as a property of a pair of directions and a linear intervention model, and prove that mutual orthogonality of $\{r_1,\ldots,r_N\}$ is sufficient for pairwise independent editability of all $N$ features simultaneously (not just two at a time).

6. Given the packing bound referenced in Exercise 4 (exponentially many near-orthogonal directions exist in $\mathbb{R}^d$ for a fixed small tolerance $\epsilon$), derive an approximate bound on the total cross-talk a single feature $f_i$ experiences from all other $N-1$ features simultaneously intervened upon with independent random signs/magnitudes $\alpha_j$, assuming each pairwise dot product is bounded by $\epsilon$ in absolute value and the $\alpha_j$ are independent, mean-zero, unit-variance.

Key papers

• Nanda, Lee & Wattenberg (2023). Emergent Linear Representations in World Models of Self-Supervised Sequence Models. arXiv:2309.00941 — a landmark case study for Property 5 above: a board-game world model (OthelloGPT) previously reported as non-linearly encoded turns out to be near-perfectly linear once probed in the model’s own relative (MINE/YOURS) frame rather than the human-preconceived absolute (BLACK/WHITE) one, and the resulting linear directions are causally validated via simple multi-layer vector addition — see othello-relative-frame-direction.
• Elhage, Hume, Olsson et al. (2022). Toy Models of Superposition. — origin of the superposition hypothesis this page’s Property 4 and Exercises 4-6 build on.
• Garg, Kleinberg & Peng (2026). How Many Features Can a Language Model Store Under the Linear Representation Hypothesis? arXiv:2602.11246 — formalizes LRH as two separable claims, linear representation ($f(\ell)=Az$) and linear accessibility (features read out by a linear probe $b_i$, not merely some function), and proves these are genuinely different in capacity: $d=O_\epsilon(k^2\log m)$ neurons suffice to both linearly represent and linearly recover any $k$-sparse set of $m$ features (exponential superposition, confirming Elhage et al.’s hypothesis within a formal framework), but $d = \Omega_\epsilon(\frac{k^2}{\log k}\log\frac{m}{k})$ is necessary once recovery is restricted to be linear — a provably larger requirement than the $O(k\log(m/k))$ that classical (non-linear-decoder) compressed sensing needs for the same task, using compressed-sensing-capacity- bounds. The paper also shows this “clean, mutually orthogonal direction per feature” picture (Property/Exercise 4-6 above) is not forced by the capacity bound alone: a feature’s representation direction and its own probe direction can be nearly orthogonal to each other, while many different features’ representation directions (or probe directions) can be highly mutually correlated, all while still achieving accurate linear recovery — the orthogonal-per-feature picture is recovered only once representation and probe vectors are additionally constrained to unit norm. Finally, adding a bias and a monotonic non-linear activation to the probe does not asymptotically raise the bound, so a single non-linear readout layer is no more capacity-efficient than a purely linear one.
• Kniazev & Fijalkow (2026). Transformers Linearly Represent Highly Structured World Models. arXiv:2605.18847 — trains an 8-layer transformer on Sudoku solving traces and finds that its linear world model is organized around the puzzle’s constraint algebra (rows/columns/boxes) rather than the surface grid cells a human would naturally decompose the board into: 243 binary probes for “is digit $d$ present in substructure $S$” reach perfect exact-match accuracy in mid-layers, while 81 per-cell digit-identity probes plateau at 80% top-1 accuracy — see Constraint-Algebra Basis Hypothesis, a new hypothesis this paper motivates about which unit a linear world model’s basis tracks when a task’s surface decomposition and its algebraic constraint structure genuinely diverge.
• Li, Michaud, Baek, Engels, Sun & Tegmark (2024). The Geometry of Concepts: Sparse Autoencoder Feature Structure. Entropy 27(4):344, arXiv:2410.19750 — directly elaborates Property 3’s parallelogram prediction: raw SAE/activation quadruplets for a genuine semantic relation often fail to form clean parallelograms because a high-variance distractor direction (e.g. word length) dominates the difference vectors; projecting it out via Linear Discriminant Analysis (LDA) recovers the structure, and generalizes the parallelogram itself to a trapezoid — one shared function-vector direction without requiring an identical displacement magnitude across instances.
• Walker, Humayun, Balestriero & Baraniuk (2026). The Linear Centroids Hypothesis: Features as Directions Learned by Local Experts. arXiv:2604.11962 — proposes a rival account, Linear Centroids Hypothesis: features are linear directions not in raw activation space but in the space of centroids (Jacobian row-sums summarizing a sub-component’s actual local input-output map). Proves linear separability in activation space is necessary but not sufficient for a genuine feature (a ReLU-collapse counterexample can make two sets linearly separable as a pure artifact of piecewise-affine folding), and finds centroid-based SAE dictionaries, mass-mean probes and PCA directions transfer more reliably across real ResNet50/DTD, DINOv2/DINOv3, GPT2-Large and Llama-3.1-8B than the same tools applied to raw activations — see walker-etal-2026-jacobian-centroid-features-separate-and-transfer-more-reliably-than-raw-activation-features-across-real-resnet50-dinov2-dinov3-gpt2-large-and-llama-3.1-8b.