In LLaVA-NeXT and OmniFusion (both Mistral-7B-based), visual and text token representations differ in linearity, intrinsic dimension and anisotropy across layers

- Mussabayev & Mussabayev analyse 16 ScienceQA test examples (1563 text and 21964 image token embeddings, 4096-d, 32 layers) in two Mistral-7B-backbone multimodal LLMs, LLaVA-NeXT 1.6 and OmniFusion, with a TwoNN-style intrinsic-dimension estimate, an SVD-based anisotropy score (share of variance in the top singular value) and a layer-to-layer linear-fit (Procrustes-style) score; the LLaVA-NeXT checkpoint is mapped to llava-next-7b approximately. [mussabayev-etal-2026-mllm-microscope-geometry-of-visual-and-text-tokens-in-multimodal-llms] - Text-token ID drops sharply within the first few layers and stabilises around 20 (slightly lower for LLaVA-NeXT, higher for OmniFusion); OmniFusion image-token ID stays consistently higher than LLaVA-NeXT's across layers. [mussabayev-etal-2026-mllm-microscope-geometry-of-visual-and-text-tokens-in-multimodal-llms] - Text-token anisotropy is high in initial layers and declines smoothly toward the final layers in both models without a bell-shaped profile; LLaVA-NeXT image tokens follow a bell-shaped curve while OmniFusion image tokens stay uniformly low. [mussabayev-etal-2026-mllm-microscope-geometry-of-visual-and-text-tokens-in-multimodal-llms] - Both models show near-perfectly linear layer-to-layer transformations; the authors attribute differences to the fusion adapter as a hypothesis ("we hypothesize"), on a single dataset and two models. [mussabayev-etal-2026-mllm-microscope-geometry-of-visual-and-text-tokens-in-multimodal-llms]