Mass-Editing Memory (MEMIT) — Multi-Layer Batch Associative-Memory Editing

methods / Causal Validation / Mass-Editing Memory (MEMIT) — Multi-Layer Batch Associative-Memory Editing

Mass-Editing Memory (MEMIT) — Multi-Layer Batch Associative-Memory Editing

Techniqueadvanced

Generalizes ROME's single rank-one weight update to a batch least-squares update that inserts thousands of (key, value) associations simultaneously, spreading the residual evenly across a whole range of MLP layers rather than writing into one layer alone — still the same linear-associative-memory / outer-product direction mechanism, just solved jointly over many facts and apportioned across depth.

Used in (2 observations)

Knowledge editing shatters transformers' natural cyclic manifold geometrymeasured

structure: Circle · models: Custom Transformer (trained from scratch on a structured synthetic knowledge graph), Llama-3-8B-Instruct, Mamba-2.8B, Llama-3.1-405B-Instruct, GPT-2-small, Mistral-7B · paper: Representation Shattering in Transformers: A Synthetic Study with Knowledge Editing

MEMIT batch-edits many facts across MLP layers via least squaresmeasured

structure: Linear Direction · models: GPT-J-6B, GPT-NeoX-20B · paper: Mass-Editing Memory in a Transformer