MEND transforms rank-one gradient factors with a hypernetwork

measured in 1 paper

Mitchell et al. note the per-example weight gradient of a dense layer is a sum of rank-one outer products (upstream gradient times input activation) [mitchell-etal-2022-mend] They train small per-layer hypernetworks that transform each rank-one factor independently, then reapply the same outer-product update with a learned step size [mitchell-etal-2022-mend] Every stage is a rank-one/outer-product direction construct, the same trigger/value pattern as ROME and MEMIT but reached by a learned transformation [mitchell-etal-2022-mend] Tested on GPT-Neo-2.7B, GPT-J-6B, T5-XL, and T5-11B, MEND reaches 0.81-0.89 edit success with low perplexity drawdown, staying effective at 11B where baselines fail [mitchell-etal-2022-mend] It uses rather than finds geometry, focusing purely on the algebra of the edit mechanism [mitchell-etal-2022-mend]

Structure

Linear Direction

Context

per-example weight gradient as a rank-one outer product (backprop identity), hypernetwork transforming each rank-one factor independently, preserving outer-product form, USES not FINDS (no claim about pre-existing geometry, focus purely on edit-mechanism algebra), scale robustness up to 11B parameters where baselines fail or cannot run

Confirmed in models

GPT-Neo-2.7B · GPT-Neo, GPT-J-6B · GPT-J, T5-XL · T5, T5-11B · T5

Method

MEND — Model Editor Networks using Gradient Decomposition

Papers

Fast Model Editing at Scale — Mitchell, Eric, Lin, Charles, Bosselut, Antoine, Finn, Chelsea, Manning, Christopher D.2022 · arXiv:2110.11309