Morphological relations are truly linear; semantic ones need an affine bias

measured in 1 paper

Xia & Kalita decompose Hernandez et al. affine LRE into a multiplicative Jacobian-only (linear) term and an additive bias term, applying both to 40 BATS relation categories on GPT-J and Llama-2-7B [xia-kalita-2025-linear-relational-decoding-of-morphology] Across 14 morphology relations the bias-free linear LRE reaches 90% top-1 faithfulness, near the full affine LRE 95% [xia-kalita-2025-linear-relational-decoding-of-morphology] For semantic/encyclopedic relations the linear LRE drops to 40% versus the affine LRE 61%, and additive-only approximators fail on morphology, confirming the Jacobian term is necessary [xia-kalita-2025-linear-relational-decoding-of-morphology] The dissociation replicates across architecture (GPT-J vs Llama-2) and 8 typologically diverse languages; no causal intervention is performed [xia-kalita-2025-linear-relational-decoding-of-morphology]

Structure

Linear Subspace

Context

decomposition of affine LRE into separate multiplicative (Jacobian-only, through-origin linear) and additive (bias-only) components, category-dependent linear-vs-affine dissociation, 90%/95% faithfulness for morphology vs. 40%/61% for semantic-encyclopedic relations, cross-architecture replication (GPT-J's parallel attention/MLP vs. Llama-2's sequential blocks), cross-typological replication across 8 languages including agglutinative Hungarian and Turkish, necessity (not just sufficiency) of the multiplicative Jacobian term for morphological relations, shown via additive-only approximator failure

Confirmed in models

GPT-J-6B · GPT-J, Llama-2-7B · Llama

Method

Linear Relational Embedding (LRE)

Papers

Linear Relational Decoding of Morphology in Language Models — Xia, Eric, Kalita, Jugal2025 · arXiv:2507.14640