Factual recall is an additive motif across component types

Chughtai, Cooney & Nanda use Direct Logit Attribution on Pythia-2.8B (primary; validated on GPT2-XL, GPT-J and other Pythia sizes including 6.9B) to dissect factual recall [chughtai-cooney-nanda-2024-summing-up-the-facts-additive-mechanisms-behind-factual-recall-in-llms] The mechanism is an "additive motif": several qualitatively different component types -- subject heads, relation heads, mixed heads, and MLPs -- independently contribute and additively sum, constructively interfering on the correct attribute [chughtai-cooney-nanda-2024-summing-up-the-facts-additive-mechanisms-behind-factual-recall-in-llms] Heads are sorted by an "attention ratio" (DLA to subject vs relation tokens), with strongly specialized values (e.g. up to 107.23 for one head) [chughtai-cooney-nanda-2024-summing-up-the-facts-additive-mechanisms-behind-factual-recall-in-llms] The framing is additivity across component types, not a low-rank per-attribute subspace (the low-rank property is only the generic fact that any attention head reads and writes a low-rank slice of the residual stream) [chughtai-cooney-nanda-2024-summing-up-the-facts-additive-mechanisms-behind-factual-recall-in-llms]