Centroid-based concept vectors for colors and color-shape conjunctions causally steer perception in three open-weight VLMs, while naive probe directions largely fail

Savietto et al. build concept vectors v in R^d (d = 3584 / 4096 / 3840) from class centroids minus the global mean of vision-token embeddings in Qwen2.5-VL 7B, InternVL2.5 8B and Gemma 3 12B, and steer by removing the component along A and adding it along B [savietto-etal-2026-the-geometry-of-representational-failures-in-vision-language-models] Color steering on 60 real photos (300 operations) succeeds 84.7% / 88.4% / 95.7% (Qwen / InternVL / Gemma) with centroid vectors, versus 3.7% / 61.1% / 3.7% for raw probe directions [savietto-etal-2026-the-geometry-of-representational-failures-in-vision-language-models] For 36 color-shape conjunctions on synthetic multi-object images, centroid vectors reach 78.1% (Qwen) and 75.3% (Gemma) under a strict remove-A, insert-B, keep-C criterion, but only 35.9% for InternVL, where PCA-regularized probes do better (46.8%) [savietto-etal-2026-the-geometry-of-representational-failures-in-vision-language-models] Shape steering on natural COCO images is weaker (centroid 19.7% / 1.1% / 25.8%) and depends strongly on source and target shape [savietto-etal-2026-the-geometry-of-representational-failures-in-vision-language-models]