Function vector extraction (causal-mediation head selection + sum)

methods / Direction Extraction / Function vector extraction (causal-mediation head selection + sum)

Function vector extraction (causal-mediation head selection + sum)

Techniqueadvanced

Identifies the small set of attention heads with the highest average indirect effect (via causal mediation / activation patching) on a diverse set of in-context-learning tasks, then sums those heads' mean per-task output activations into a single 'function vector' representing the task.

Used in (8 observations)

A last-token function vector transplants across languages in two stagesmeasured

structure: Linear Direction · models: XGLM-7.5B, EuroLLM-9B, mT5-xl · paper: How Do Multilingual Language Models Remember Facts?

Multimodal function vectors causally raise visual-relation accuracymeasured

structure: Linear Direction · models: OpenFlamingo-4B · paper: Multimodal Function Vectors for Visual Relations

A function vector decomposes additively into causal per-shot sub-vectorsmeasured

structure: Linear Direction · models: Gemma-2-2B, Gemma-2-9B, Gemma-2-27B, Llama-3.2-1B, Llama-3.2-3B, Llama-3.1-8B-Instruct · paper: How Few-Shot Examples Add Up: A Causal Decomposition of Function Vectors in In-Context Learning

Function vectors are compact directions that causally trigger ICL tasksmeasured

structure: Linear Direction · models: GPT-J-6B, GPT-NeoX-20B, Llama-2-7B, Llama-2-13B, Llama-2-70B, LLaMA-7B, LLaMA-13B, LLaMA-30B, Pythia-2.8B, Pythia-6.9B, Pythia-12B · paper: Function Vectors in Large Language Models, In-Context Learning Creates Task Vectors, Unifying Attention Heads and Task Vectors via Hidden State Geometry in In-Context Learning, Do Different Prompting Methods Yield a Common Task Representation in Language Models?

Task vectors are locally realized at certain tokens despite decodable identitymeasured

structure: Linear Direction · models: Gemma 3 4B, Qwen3-8B · paper: Just-in-Time and Distributed Task Representations in Language Models

Function vectors steer even where logit lens and probes cannot decodemeasured

structure: Linear Direction · models: Llama-3.1-8B, Llama-3.1-8B-Instruct, Gemma-2-9B, Gemma-2-9B-it, Mistral-7B-v0.3, Mistral-7B-Instruct-v0.3 · paper: Steerable but Not Decodable: Function Vectors Operate Beyond the Logit Lens

ICL separability rises first, then alignment with the unembeddingmeasured

structure: Linear Direction, Linear Separability · models: Llama-2-7B, Llama-2-13B, Llama-2-70B, Llama-3-8B, Llama-3-70B, Gemma-2B, Gemma-7B · paper: Unifying Attention Heads and Task Vectors via Hidden State Geometry in In-Context Learning

Function-vector heads drive ICL; induction heads matter little and FV heads evolve from themmeasured

structure: Linear Direction · models: Pythia-70M, Pythia-160M, Pythia-410M, Pythia-1B, Pythia-1.4B, Pythia-2.8B, Pythia-6.9B, GPT-2-small, GPT-2-Medium, GPT-2-Large, GPT-2-XL, Llama-2-7B · paper: Which Attention Heads Matter for In-Context Learning?