A single attention head writes a language-independent subject-number signal into a one-dimensional (PC1) residual-stream direction, causally verified across English and Spanish

- Ferrando & Costa-jussà find, via activation patching and direct logit attribution on Gemma 2B, that attention head L13H7 writes a "subject number" signal to the last residual-stream position, which a downstream MLP neuron (neuron 2069 in MLP13, among others) reads; PCA on this head's output shows PC1 clearly distinguishes singular from plural subjects, which the authors read as the head writing "into a 1-dimensional subspace where the subject number signal is encoded". [ferrando-costajussa-2024-shared-agreement-direction] - The circuit is described as highly consistent between English and Spanish subject-verb-agreement prompts; steering the head output at the last position with the unit-norm PC1 found on English examples (PC1_English, coefficient chosen on a validation set) flips Spanish predicted verb number from singular to plural and from plural to singular, with other top predicted tokens mostly unchanged. Only English and Spanish are tested. [ferrando-costajussa-2024-shared-agreement-direction] - A similar attention head is reported in Gemma 2 2B (L19H3) and Gemma 7B (L22H6) from activation patching and PCA; steering with PC1_English is reported for Gemma 2 2B only, where it significantly reduces the Spanish logit difference and completely flips the plural case, and no steering is reported for Gemma 7B. The paper does not state whether base or instruction-tuned checkpoints were used. [ferrando-costajussa-2024-shared-agreement-direction]