MATH · IN · MODELS
methods / Causal Validation / Rank-One Model Editing (ROME) / Linear Associative Memory Editing

Rank-One Model Editing (ROME) / Linear Associative Memory Editing

Techniqueadvanced

Treats an MLP's down-projection weight matrix as a linear associative memory solving W·K≈V, then inserts one new (key, value) association with a minimal closed-form rank-one update — mathematically an outer-product direction mechanism (a whitened key direction detects the trigger, a value direction is written to the output), but the injected value is optimized fresh rather than extracted from any pre-existing linear geometry the model already had for that fact. USES the key-direction/value-injection structure as an editing tool; does not itself claim the edited fact was linearly encoded in the model beforehand.

Used in (4 observations)

structure: Linear Direction · models: GPT-J-6B, GPT-2 XL · paper: Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models
structure: Linear Subspace · models: GPT-2-small, GPT-2-Medium, GPT-2-Large, GPT-J-6B, Pythia-1B, Pythia-2.8B, Pythia-6.9B, Llama-2-7B, Llama-2-13B, Llama-3-8B, Llama-3.1-8B · paper: Knowledge in Superposition: Unveiling the Failures of Lifelong Knowledge Editing for Large Language Models
structure: Circle · models: Custom Transformer (trained from scratch on a structured synthetic knowledge graph), Llama-3-8B-Instruct, Mamba-2.8B, Llama-3.1-405B-Instruct, GPT-2-small, Mistral-7B · paper: Representation Shattering in Transformers: A Synthetic Study with Knowledge Editing
structure: Linear Direction · models: GPT-2 XL, GPT-J-6B · paper: Locating and Editing Factual Associations in GPT