ICL task-recognition and task-learning heads project onto a label subspace

measured in 1 paper

Yang, Cho & Inoue introduce Task Subspace Logit Attribution, scoring each attention head output by projection onto the span of label-token unembedding vectors [yang-cho-inoue-2026-task-subspace-attention-heads] Task-recognition heads align with the task subspace while task-learning heads rotate the hidden state toward the correct-minus-incorrect label direction within it [yang-cho-inoue-2026-task-subspace-attention-heads] Across Llama3-8B layers, hidden-update vs TR-head subspace alignment correlates rho=0.94, and logit-difference vs TL-head alignment rho=0.53 [yang-cho-inoue-2026-task-subspace-attention-heads] Injecting the summed top-3% TR-head outputs into zero-shot prompts raises classification accuracy 9.2% to 40.4% (TR-based), while TL-based vectors dominate open-ended generation; ablating each collapses its own role, tested across 6 models [yang-cho-inoue-2026-task-subspace-attention-heads]

Structure

Linear Subspace, Linear Direction

Context

task subspace logit attribution, task-recognition vs. task-learning attention heads, ICL task-vector patching into zero-shot prompts, dissociable causal head roles

Confirmed in models

Llama-3-8B · Llama, Llama-3.1-8B · Llama, Llama-3.2-3B · Llama, Qwen2-7B (base) · Qwen, Qwen2.5-32B · Qwen, Yi-34B (base) · Yi

Method

Direct Logit Attribution (DLA), Activation patching

Papers

Task Recognition and Task Learning Heads Align In-Context Hidden States with a Label-Unembedding Task Subspace — Yang, Haolin, Cho, Hakaze, Inoue, Naoya2026 · arXiv:2509.24164