MATH · IN · MODELS

LLaVA

various (academic)

Structures found in this family (3)

Linear Direction, Anisotropy, Representational-similarity trajectory across depth

By model (6)

LLaVA-1.5-7B · 7B
Anisotropy, Linear Direction, Representational-similarity trajectory across depth
LLaVA-1.5-13B · 13B
Anisotropy, Representational-similarity trajectory across depth
LLaVA-Llama-3-8B · 8B
Linear Direction
LLaVA-More (CLIP ViT-L/14-336 + Llama-3.1-8B) · 8B
Linear Direction
LLaVA-NeXT-13B · 13B
Linear Direction
LLaVA-NeXT-7B · 7B
Linear Direction

Observations (7)

SOMP-selected attention heads causally control target behavior across real unimodal and multimodal transformers
MLLM visual tokens split into sink, dead, alive clusters
Fewer than 25 register neurons with consistent directions cause ViT outlier tokens
A contrastive spatial-ID direction is causally bound into object tokens across 11 real VLMs
Oversized vision-token norms suppress VLM spatial awareness
SAE decoder directions rotate during real vision-language fine-tuning, and the most-rotated subset is causally load-bearing for spatial reasoning
MLLM visual tokens diverge from frozen VFM features; alignment loss recovers accuracy

Papers

Head Pursuit: Probing Attention Specialization in Multimodal Transformers (2025), What Do Visual Tokens Really Encode? Uncovering Sparsity and Redundancy in Multimodal Large Language Models (2026), Vision Transformers Don't Need Trained Registers (2025), Linear Mechanisms for Spatiotemporal Reasoning in Vision Language Models (2026), Beyond Semantics: Rediscovering Spatial Awareness in Vision-Language Models (2025), Towards Understanding Multimodal Fine-Tuning: A Case Study into Spatial Features (2026), Visual Representation Alignment for Multimodal Large Language Models (2025)