Transformer-Patcher corrects mistakes by adding one key-value neuron per patch

Huang et al. correct a mistake by appending one key row and one value column to the last FFN layer, adding a key direction that detects the trigger via inner product and a value direction to the output [huang-etal-2023-transformer-patcher] This is the same outer-product pattern as ROME but realized by growing the weight matrices rather than editing existing entries [huang-etal-2023-transformer-patcher] A margin-based memory loss enforces locality across many sequential patches, not any geometric constraint between different patches' key directions [huang-etal-2023-transformer-patcher] It uses rather than finds geometry, correcting thousands of sequential mistakes on BERT-base and BART-base where MEND and KnowledgeEditor collapse after only a few [huang-etal-2023-transformer-patcher]