Per-head attention-contribution probes decode relation type above full-state probes

measured in 1 paper

Popovic & Farber decompose each attention head's contribution to the residual stream via DLA-style attribution and probe these per-head features for relation type [popovic-farber-2026-tracing-relational-knowledge-recall-in-large-language-models] Linear probes decode FewRel relation type in a 5-way-5-shot task at 89.66-91.09% accuracy across Llama-3.2-1B/3B-Instruct, Llama-3.1-8B-Instruct, and Qwen3-4B-Instruct [popovic-farber-2026-tracing-relational-knowledge-recall-in-large-language-models] This exceeds probes fit on the full attention-state baseline (75.06-86.79%) on the same models and task [popovic-farber-2026-tracing-relational-knowledge-recall-in-large-language-models]

Structure

Linear Separability

Context

per-head attention-contribution decomposition as a probing feature space, more informative than the full pooled attention state, relation-specific "contrast directions" used to structure a multi-way classification probe rather than a single binary one

Confirmed in models

Llama-3.2-1B-Instruct · Llama, Llama-3.2-3B-Instruct · Llama, Llama-3.1-8B-Instruct · Llama, Qwen3-4B-Instruct · Qwen

Method

Linear probing, Direct Logit Attribution (DLA)

Papers

Tracing Relational Knowledge Recall in Large Language Models — Popović, Nicholas, Färber, Michael2026 · arXiv:2604.19934