Relation-decoding operator (affine map between representation spaces)

structures / Linear Structures / Relation-decoding operator (affine map between representation spaces)

Relation-decoding operator (affine map between representation spaces)

TOOLoperatoraffineJacobian-derived affine mapintermediatehow it's classified →

A per-relation affine map LRE(s) = βW_r s + b_r, with W_r the mean Jacobian of the model's subject-to-object computation, that reproduces the model's own relation decoding. It is a map (matrix plus offset) acting on representations, not a direction or subspace of the activation space, so it is classified as an operator.

Replicationcomputed from the corpus — never hand-assigned

2 papers · no shared authors1 architecture class1 domain4 model families · across papers

Filled = two or more values reported by papers that share no author — replication. Outlined = two or more values, but all from a single study — breadth, not replication. Grey = a single value. Derived from paper authorship and each model's architecture class, domain and family; it updates itself when a paper is added.

Definition

For a relation $r$ the model computes an object representation $o=F(s,c)$ from a subject representation $s$ in context $c$ . A first-order expansion gives the affine operator

$\mathrm{LRE}_r(s)=\beta\,W_r\,s+b_r,\qquad W_r=\mathbb E\!\left[\frac{\partial F}{\partial s}\right],$

with $W_r$ the mean Jacobian over a few examples of the relation. Unlike Linear Direction (one vector of $\mathbb R^d$ ) or Linear Subspace (a subset of $\mathbb R^d$ ), $W_r$ is a matrix mapping one representation (subject) to another (object), so the object here is the map itself.

Relative to other operator nodes

Conceptor (soft ellipsoidal region) is a PSD contraction acting on one cloud; this operator maps between two representation sets and is not required to be symmetric or contractive. Jacobian non-normality (Schur decomposition of per-layer operators) studies the per-block Jacobian across layers; here the Jacobian is averaged over examples of one relation and used as a predictor of the model’s output.

Key evidence

Relation decoding is well-approximated by a single affine map for roughly half of tested relations, but fails almost completely for others despite the model still answering correctly. Hernandez, Sharma, Haklay, Meng, Wattenberg, Andreas, Belinkov & Bau (2023) fit $\mathrm{LRE}(s)=\beta W_r s+b_r$ for 47 relations across GPT-J, GPT2-XL and LLaMA-13B: faithfulness above 60% for nearly half the relations, strong faithfulness-causality correlation (R=0.84 on GPT-J), but near-zero faithfulness (≤7%) for relations like “company CEO”. See lre-faithful-and-causal-for-half-of-relations.
The family of relation operators is highly compressible and clusters by shared coarse-grained property rather than by individual relation. Christ, Csiszárik, Becsó & Varga (2025) fit order-3 tensor networks over 47 and 79 relations (main text on GPT-J; Llama 3.1 8B and GPT-NeoX-20B in an appendix): under one million parameters beat stacked per-relation matrices (about 788 million), and cross-evaluation shows block structure driven by shared properties. The paper does not measure a shared linear subspace. See christ-etal-2025-relation-decoding-operators-cluster-by-shared-coarse-grained-property-not-relation-identity.

How to detect it

Estimate the mean Jacobian of the subject-to-object computation from a few examples of a relation, form $\beta W_r s+b_r$ , and test (i) faithfulness: does it decode to the model’s own top token, and (ii) causality: does editing $s$ via the low-rank pseudoinverse change the model’s output.

Key papers

Hernandez et al. (2023). Linearity of Relation Decoding in Transformer Language Models. arXiv:2308.09124 — introduces the LRE.
Christ et al. (2025). The Structure of Relation Decoding Linear Operators in Large Language Models. arXiv:2510.26543 — compressibility and property-level clustering of the operator family.

Found in (2 observations · 4 families)

GPT-J

The Structure of Relation Decoding Linear Operators in Large Language Models (2025)measured

The family of linear relation-decoding operators is highly compressible and clusters by shared coarse-grained property, not by individual relation identity

Details

- Christ et al. extend Hernandez et al.'s single-relation Linear Relational Embedding (LRE) decoders to collections of relations (47 in the Hernandez dataset, 79 in an extended dataset) and show that order-3 tensor networks (SimpleOrder3Network, TriangleTensorNetwork) trained end-to-end compress the collection: with under one million parameters they outperform baselines (stacked per-relation matrices, about 788 million parameters, mean faithfulness 0.41) in faithfulness; main-text results are for GPT-J, with Llama 3.1 8B and GPT-NeoX-20B in Appendix C. [christ-etal-2025-relation-decoding-operator-structure] - A cross-evaluation protocol, applying each relation's decoder to the subjects of every other relation, yields off-diagonal faithfulness above 0 (several above 0.7) and visible block structure, e.g. characteristic gender, university degree gender and occupation gender sharing the concept of gender; some overlaps are syntactic rather than semantic (Sec. 4.1-4.2, Fig. 2-3). [christ-etal-2025-relation-decoding-operator-structure] - The authors conclude the decoders extract recurring coarse-grained properties rather than distinct relations; this is their interpretation of the cross-evaluation results, and they note that "tensor networks capture structural regularities beyond the simple low-rank property of individual relations" (Sec. 4.3). The paper does not measure a shared linear subspace. [christ-etal-2025-relation-decoding-operator-structure]

models: GPT-J-6B · method: Linear Relational Embedding (LRE)

Linearity of Relation Decoding in Transformer Language Models (2023)measured

Linear relational embeddings are faithful and causal for half of relations

Details

Hernandez et al. approximate a transformer's relation-decoding function by a first-order Taylor expansion LRE(s)=beta*W_r*s+b_r, with W_r the mean Jacobian estimated from 5-8 examples [hernandez-etal-2023-lre] It is validated by faithfulness (decoded top token matches the model's own) and causality (low-rank pseudoinverse inversion redirects the real output) [hernandez-etal-2023-lre] Across 47 relations on GPT-J, GPT-2-XL, and LLaMA-13B, faithfulness exceeds 60% for nearly half the relations, with strong faithfulness-causality correlation (R=0.84 on GPT-J) [hernandez-etal-2023-lre] Relations like "company CEO" (0.06/0.31) fail almost completely despite correct model behavior, so linear relation decoding is real but heterogeneous across relation types [hernandez-etal-2023-lre]

models: GPT-J-6B · method: Linear Relational Embedding (LRE)