Edited facts' key vectors overlap in a whitened memory space

measured in 1 paper

Hu et al. extend ROME's closed-form associative-memory solution to n edits and derive an interference term equal to a whitened normalized dot-product between two facts' key vectors [hu-etal-2025-knowledge-in-superposition] Across 11 LLMs (GPT-2 family, GPT-J-6B, Pythia 1B-6.9B, Llama-2/3 family) the whitened-overlap distribution is heavy-tailed and high-kurtosis with clustered near-maximal overlaps, absent in the raw key vectors ("knowledge superposition") [hu-etal-2025-knowledge-in-superposition] Sequential ROME edits on GPT-J progressively reduce confidence in unrelated held-out facts, tying the measured overlap geometry to a lifelong-editing forgetting failure [hu-etal-2025-knowledge-in-superposition]

Structure

Linear Subspace

Context

associative-memory key-vector overlap, whitened interference distribution, knowledge superposition, lifelong knowledge editing failure

Confirmed in models

GPT-2-small · GPT, GPT-2-Medium · GPT, GPT-2-Large · GPT, GPT-J-6B · GPT-J, Pythia-1B · Pythia, Pythia-2.8B · Pythia, Pythia-6.9B · Pythia, Llama-2-7B · Llama, Llama-2-13B · Llama, Llama-3-8B · Llama, Llama-3.1-8B · Llama

Method

Rank-One Model Editing (ROME) / Linear Associative Memory Editing

Papers

Knowledge in Superposition: Unveiling the Failures of Lifelong Knowledge Editing for Large Language Models — Hu, Chenhui, Cao, Pengfei, Chen, Yubo, Liu, Kang, Zhao, Jun2025 · arXiv:2408.07413