A contrastive spatial-ID direction is causally bound into object tokens across 11 real VLMs

measured in 1 paper

Kang, Chen, Gkioxari & Perona (2026) define a canonical spatial-ID direction as a difference-of-means over grid position across 11 real VLMs (LLaVA, LLaMA, Qwen, InternVL, Gemma families), well-approximated by a fixed linear map of positional encoding (rank-3 fit, R^2 ~0.85) [kang-etal-2026-linear-mechanisms-for-spatiotemporal-reasoning-in-vlms] Activation-patching 'mirror swap' localizes the effect to object-word tokens at intermediate layers, with a color-swap control showing near-null effect [kang-etal-2026-linear-mechanisms-for-spatiotemporal-reasoning-in-vlms] Directly substituting a target spatial ID into an object token's residual stream shifts ground-truth logit-based belief 64.4% of the way to the target, versus 29.5% for a noise baseline; an analogous temporal-ID binding mechanism extends to video VLMs (LLaVA-Video, VideoLLaMA3, Qwen2.5) [kang-etal-2026-linear-mechanisms-for-spatiotemporal-reasoning-in-vlms]

Structure

Linear Direction

Context

spatial ID binding, difference-of-means direction

Confirmed in models

LLaVA-1.5-7B · LLaVA, Llama-3.1-8B-Instruct · Llama, InternVL3-8B · InternVL, Gemma-7B-it · Gemma

Method

Difference-in-means direction, Activation patching

Papers

Linear Mechanisms for Spatiotemporal Reasoning in Vision Language Models — Kang, Raphi, Chen, Hongqiao, Gkioxari, Georgia, Perona, Pietro2026 · arXiv:2601.12626