Few-shot integer-addition ability in Llama-3-8B-Instruct reduces to three attention heads with six-dimensional activation subspaces, where four dimensions track the unit digit with trigonometric functions of periods 2, 5, and 10 and two dimensions track magnitude with low-frequency components

- Hu et al. localize the few-shot "add k" mechanism of Llama-3-8B-Instruct (main body; five-shot prompts, k in [1,30], inputs in [1,100]) to three attention heads, (15,2), (15,1) and (13,6), selected by a sparse optimization over all 1,024 heads followed by ablations; mean-ablating the three heads drops five-shot accuracy from 0.87 to 0.43. [hu-etal-2025-icl-addition-activation-subspaces] - PCA of each head's 128-dimensional head vectors across tasks k finds a six-dimensional subspace (six PCs explain 97% of the task variance); a linear rotation gives feature directions fitting trigonometric functions of k with periods 2, 5, 10, 10, 25 and 50, so four dimensions (periods 2, 5, 10) track the units digit and two (periods 25, 50) track coarse magnitude (the tens digit); projecting onto or out of the unit-digit subspace is used as a causal check. In the appendix, Phi-4-14B shows similar structure (all three main heads fit the six trigonometric patterns; 95% of variance needs 7-8 PCs), while for Qwen-2.5-7B only two of the four main heads, (21,0) and (21,4), fit the periodic unit-digit patterns well (5-8 PCs); the authors conjecture tokenization (digit-by-digit numbers). [hu-etal-2025-icl-addition-activation-subspaces] - They also derive a mathematical identity relating the heads' aggregator and extractor subspaces, and use it to show that label tokens carry 96-99% of each main head's task signal and that each demonstration contributes a signal y_i - x_i. [hu-etal-2025-icl-addition-activation-subspaces]