Linear Relational Embedding (LRE)

methods / Direction Extraction / Linear Relational Embedding (LRE)

Techniqueintermediate

Approximates a transformer's subject-to-object relation decoding F(s,c) by a first-order Taylor expansion — the mean Jacobian W and bias b of F with respect to the subject representation s, estimated from a handful of examples — yielding a single affine map LRE(s)=βWs+b that is faithful (matches the model's real next-token prediction) and causal (inverting it edits which object the model predicts) for roughly half of tested relations, but conspicuously fails for others despite the model still answering those relations correctly.

Used in (5 observations)

ICL answer tokens are affine functions in a PCA subspacemeasured

structure: Linear Direction · models: GPT-J-6B, Llama-3.1-8B, OLMo 2 32B, Pythia-12B, Gemma 3 27B Instruct, GPT-NeoX-20B · paper: Functional Subspace, where language models can use vector algebra to solve problems

Linear relational embeddings are faithful and causal for half of relationsmeasured

structure: Linear Direction · models: GPT-J-6B, GPT-2 XL, LLaMA-13B · paper: Linearity of Relation Decoding in Transformer Language Models

LRE affine-map linearity scores correlate with hallucination rate across four real instruction-tuned LLMsmeasured

structure: Linear Direction · models: Gemma-7B-it, Llama-3.1-8B-Instruct, Mistral-7B-Instruct-v0.3, Qwen2.5-7B-Instruct · paper: Relational Linearity is a Predictor of Hallucinations

Factual recall is linear superposition with a proven dimension boundmeasured

structure: Linear Subspace · models: Qwen2.5-0.5B, Qwen3-14B, Llama-3.1-8B, Llama-3.2-1B, Phi-4 (15B) · paper: Geometric Factual Recall in Transformers

Morphological relations are truly linear; semantic ones need an affine biasmeasured

structure: Linear Subspace · models: GPT-J-6B, Llama-2-7B · paper: Linear Relational Decoding of Morphology in Language Models