MLLM visual tokens diverge from frozen VFM features; alignment loss recovers accuracy

measured in 1 paper

Yoon et al. measure CKNNA similarity between LLaVA-1.5-7B/13B's internal visual-token hidden states and frozen vision-foundation-model features (DINOv2, CLIP, SAM, Depth Anything V2, RADIO) across all 32 layers [yoon-etal-2025-viral-visual-representation-alignment] Similarity drops sharply after the early layers and only partially recovers in the middle, so the MLLM's visual processing diverges from the pretrained encoder's geometry with depth [yoon-etal-2025-viral-visual-representation-alignment] Adding an explicit cosine-alignment loss toward the frozen VFM causally improves benchmarks (LLaVA-1.5-7B+DINOv2: MMVP 28.20 to 33.33, POPE 85.70 to 88.32, MME 1650.21 to 1694.52) [yoon-etal-2025-viral-visual-representation-alignment] A permutation-sensitivity test shows the aligned model is more spatially sensitive (patch-shuffling degrades accuracy 13.0% with the loss vs 6.5% without) [yoon-etal-2025-viral-visual-representation-alignment]

Structure

Representational-similarity trajectory across depth

Context

CKNNA cross-model representational alignment, training-time alignment-loss intervention, permutation-sensitivity validation

Confirmed in models

LLaVA-1.5-7B · LLaVA, LLaVA-1.5-13B · LLaVA

Method

CKNNA representational alignment, Representation-alignment loss regularization

Papers

Visual Representation Alignment for Multimodal Large Language Models — Yoon, Heeji, Jung, Jaewoo, Kim, Junwan, Choi, Hyungyu, Shin, Heeseong, Lim, Sangbeom, An, Honggyu, Kim, Chaehyun, Han, Jisang, Kim, Donghyun, Eom, Chanho, Hong, Sunghwan, Kim, Seungryong2025 · arXiv:2509.07979