Factual recall is linear superposition with a proven dimension bound

measured in 1 paper

Ravfogel et al. model subject embeddings as a sum of per-attribute vectors read by a relation-conditioned ReLU-gated MLP, proving (Theorem 4.1) a 1-layer transformer plus 3-layer MLP solves single-hop recall for N entities and R relations when d=4R*log(N)+1 [ravfogel-etal-2026-geometric-factual-recall-transformers] Synthetic experiments confirm the scaling: trainable embeddings memorize once d>=128 while frozen embeddings need d>=512 for R=16 [ravfogel-etal-2026-geometric-factual-recall-transformers] On five real LMs a rank-512 affine probe recovers the LM-head output embedding (best-layer MRR 0.44-0.69 across entity categories), reversing Hernandez et al. reported output-side non-linearity [ravfogel-etal-2026-geometric-factual-recall-transformers] Minimum-norm subject-embedding perturbations swap the queried attribute with high selectivity, and a frozen relation-selector MLP transfers zero-shot to unseen relation bijections [ravfogel-etal-2026-geometric-factual-recall-transformers]

Structure

Linear Subspace

Context

proven embedding-dimension bound d=4R log(N)+1 for superposition-based single-hop relational recall (Theorem 4.1), linear superposition of per-relation attribute vectors in subject embeddings, read out by a relation-conditioned ReLU-gated MLP selector, rank-512 low-rank affine probe recovering LM-head output-embedding row D[t_o] via cosine loss, across 5 real pretrained LMs and 6 entity categories, minimum-norm causal embedding perturbations that swap the queried attribute with high selectivity, zero-shot transfer of a frozen relation-selector MLP to novel relation bijections, theoretical extension of, and partial reversal of a specific non-linearity finding within, Hernandez et al. (2023)'s Linear Relational Embedding (LRE) construction

Confirmed in models

Qwen2.5-0.5B · Qwen, Qwen3-14B · Qwen, Llama-3.1-8B · Llama, Llama-3.2-1B · Llama, Phi-4 (15B) · Phi

Method

Linear Relational Embedding (LRE), Linear probing

Papers

Geometric Factual Recall in Transformers — Ravfogel, Shauli, Yehudai, Gilad, Bruna, Joan, Bietti, Alberto2026 · arXiv:2605.12426