MATH · IN · MODELS

LLaVA

various (academic)

Structures found in this family (4)

Linear Direction, Anisotropy, Intrinsic-dimension profile across depth, Representational-similarity trajectory across depth

By model (6)

LLaVA-1.5-7B · 7B
Anisotropy, Linear Direction, Representational-similarity trajectory across depth
LLaVA-NeXT-7B · 7B
Linear Direction, Intrinsic-dimension profile across depth, Anisotropy
LLaVA-1.5-13B · 13B
Anisotropy, Representational-similarity trajectory across depth
LLaVA-Llama-3-8B · 8B
Linear Direction
LLaVA-More (CLIP ViT-L/14-336 + Llama-3.1-8B) · 8B
Linear Direction
LLaVA-NeXT-13B · 13B
Linear Direction

Observations (8)

SOMP-selected attention heads causally control target behavior across real unimodal and multimodal transformers
MLLM visual tokens split into sink, dead, alive clusters
In LLaVA-NeXT and OmniFusion (both Mistral-7B-based), visual and text token representations differ in linearity, intrinsic dimension and anisotropy across layers
Fewer than 25 register neurons with consistent directions cause ViT outlier tokens
A contrastive spatial-ID direction is causally bound into object tokens across 11 real VLMs
Oversized vision-token norms suppress VLM spatial awareness
SAE decoder directions rotate during real vision-language fine-tuning, and the most-rotated subset is causally load-bearing for spatial reasoning
MLLM visual tokens diverge from frozen VFM features; alignment loss recovers accuracy

Papers

Head Pursuit: Probing Attention Specialization in Multimodal Transformers (2025), What Do Visual Tokens Really Encode? Uncovering Sparsity and Redundancy in Multimodal Large Language Models (2026), MLLM-Microscope: Unlocking Hidden Structure Within Multimodal Large Language Models (2026), Vision Transformers Don't Need Trained Registers (2025), Linear Mechanisms for Spatiotemporal Reasoning in Vision Language Models (2026), Beyond Semantics: Rediscovering Spatial Awareness in Vision-Language Models (2025), Towards Understanding Multimodal Fine-Tuning: A Case Study into Spatial Features (2026), Visual Representation Alignment for Multimodal Large Language Models (2025)