Oversized vision-token norms suppress VLM spatial awareness

measured in 1 paper

- In LLaVA-1.5-7B, vision-token L2 norms are 1-2 orders of magnitude (up to 3) larger than text-token norms, so vision tokens dominate attention logits. [qi-etal-2025-beyond-semantics-spatial-awareness-vlms] - These oversized norms suppress positional sensitivity, making the model behave like a bag of tokens (shown by the Token Permutation Test and Spatial Compression Study). [qi-etal-2025-beyond-semantics-spatial-awareness-vlms] - RMS-normalizing vision embeddings toward the text-norm scale restores spatial awareness (+2.67 on the 2D spatial task from norm alone, +8.17 with multilayer features; VQAv2 +0.97, CV-Bench-2D +3.32). [qi-etal-2025-beyond-semantics-spatial-awareness-vlms] - Diagnostics are the Token Permutation Test, Spatial Compression Study, Embedding Norm Analysis, Attention Entropy and Attention Visualization. [qi-etal-2025-beyond-semantics-spatial-awareness-vlms]

Structure

Anisotropy

Context

vision-token embedding norms are 10-100x larger than text-token norms in a real VLM, oversized vision-embedding norms suppress positional (RoPE) sensitivity, RMS renormalization of vision embeddings causally restores spatial-reasoning accuracy

Confirmed in models

LLaVA-1.5-7B · LLaVA

Method

Geometric analysis, Causal interventions (steering)

Papers

Beyond Semantics: Rediscovering Spatial Awareness in Vision-Language Models — Qi, Jianing, Liu, Jiawei, Tang, Hao, Zhu, Zhigang2025 · arXiv:2503.17349