A function vector decomposes additively into causal per-shot sub-vectors

measured in 1 paper

Wang et al. redefine the function vector per-prompt and decompose it into additive per-example sub-vectors extracted via attention masking [wang-etal-2026-causal-decomposition-function-vectors] Global OLS fits weights so v_FV = sum_i w_i v_i + epsilon, reconstructing closely (mean cosine >=0.925, R^2 >=0.875) [wang-etal-2026-causal-decomposition-function-vectors] Mismatched-dictionary and orthogonalized-sub-FV null controls collapse to much lower fit, confirming the decomposition is not vacuous [wang-etal-2026-causal-decomposition-function-vectors] Injecting the reconstructed vector into 0-shot prompts recovers most of the full vector's causal steering effect (accuracy ratio 0.818-1.116) [wang-etal-2026-causal-decomposition-function-vectors] Under contextualization, attention shifts its share toward unambiguous examples (32% to 61%), with the query-key pathway dominating the gain [wang-etal-2026-causal-decomposition-function-vectors]

Structure

Linear Direction

Context

per-prompt function vector (sum of FV-head activations, not cross-prompt average), attention-masked per-example sub-function-vector extraction, OLS additive reconstruction (cosine >=0.925, R^2>=0.875) with null-control comparisons, causal injection-accuracy recovery of the reconstructed vector (ratio 0.818-1.116), attention-reweighting of sub-vector contributions under contextualization (Shapley QK vs. V decomposition)

Confirmed in models

Gemma-2-2B · Gemma, Gemma-2-9B · Gemma, Gemma-2-27B · Gemma, Llama-3.2-1B · Llama, Llama-3.2-3B · Llama, Llama-3.1-8B-Instruct · Llama

Method

Function vector extraction (causal-mediation head selection + sum), Causal interventions (steering), Activation patching

Papers

How Few-Shot Examples Add Up: A Causal Decomposition of Function Vectors in In-Context Learning — Wang, Entang, Wang, Yiwei, Bakalova, Aleksandra, Hahn, Michael2026 · arXiv:2605.16591