Copy-suppression: a head anti-copies whatever token it would predict

McDougall et al. compose GPT-2 Small head 10.7's OV circuit with the model's embedding and unembedding into an explicit vocab-by-vocab matrix [mcdougall-etal-2023-copy-suppression-comprehensively-understanding-an-attention-head] Its diagonal, each token's self-projection through the circuit, is among the top-10 most negative entries in 84.70% of vocabulary columns [mcdougall-etal-2023-copy-suppression-comprehensively-understanding-an-attention-head] The composed QK circuit's diagonal is the largest entry in its row for 95.72% of tokens, so the head attends to then anti-copies whichever token it would otherwise predict [mcdougall-etal-2023-copy-suppression-comprehensively-understanding-an-attention-head] A projection-only ablation (CSPA) onto the per-token negative-unembedding direction preserves 76.9-95.2% of the head's effect, confirming the geometric claim is causally load-bearing [mcdougall-etal-2023-copy-suppression-comprehensively-understanding-an-attention-head]