VLMs encode chart signal linearly but do not route it to decisions

measured in 1 paper

Kumar et al. train layer-wise logistic-regression probes on hidden states of Qwen2.5-VL-7B/32B-Instruct and InternVL3-8B for chart-claim verification [kumar-etal-2026-encoded-but-not-routed-explaining-the-table-chart-gap-in-scientific-claim-verification] On InternVL3-8B the mean-pool probe reaches 79.0% accuracy while the model's own decision reaches only 56.2%; on Qwen2.5-VL-32B probe AUROC is 88.9% versus 71.9% at the last token (McNemar p<0.01) [kumar-etal-2026-encoded-but-not-routed-explaining-the-table-chart-gap-in-scientific-claim-verification] An attention analysis finds the Qwen models attend to image tokens at only 4-11% of a proportional baseline at the final layer, while InternVL3 attends at about 93% [kumar-etal-2026-encoded-but-not-routed-explaining-the-table-chart-gap-in-scientific-claim-verification] The authors note the analysis is diagnostic, not causal [kumar-etal-2026-encoded-but-not-routed-explaining-the-table-chart-gap-in-scientific-claim-verification]

Structure

Linear Separability

Context

a probe-vs-decision accuracy gap as direct quantified evidence that information is linearly present in intermediate representations but not used by the model's own downstream decision, layer-wise last-token vs. mean-pool probing to localize where a signal is best linearly recoverable

Confirmed in models

Qwen2.5-VL 7B Instruct · Qwen, Qwen2.5-VL 32B Instruct · Qwen, InternVL3-8B · InternVL

Method

Linear probing

Papers

Encoded but Not Routed: Explaining the Table-Chart Gap in Scientific Claim Verification — Kumar, Sunisth, Ho, Xanh, Schopf, Tim, Greiner-Petter, Andre, Boudin, Florian, Aizawa, Akiko2026 · arXiv:2606.01679