FFN value vectors are vocabulary-space directions in upper layers

Geva et al. reframe the FFN sublayer as an unnormalized key-value memory FF(x)=f(x.K^T).V, treating each value row as a residual-space direction [geva-etal-2020-ffn-key-value-memories] Projecting each value vector through the output embedding yields a vocabulary distribution whose top token increasingly agrees with the paired key's next token with depth (near 0% early to 3.5% by layer 16) [geva-etal-2020-ffn-key-value-memories] Human annotation finds a recognizable trigger pattern for every key (3.6 patterns/key), with shallow n-gram patterns in lower layers and semantic ones in upper layers [geva-etal-2020-ffn-key-value-memories] Each FFN output is a compositional weighted sum of hundreds of active memory cells, refining the residual stream by often vetoing toward a compromise token [geva-etal-2020-ffn-key-value-memories] This anticipates the logit-lens and the FFN-editing basis later exploited by ROME and MEMIT [geva-etal-2020-ffn-key-value-memories]