MATH · IN · MODELS

SigLIP

Google DeepMind

Structures found in this family (4)

Dimensional collapse, Linear Direction, Anisotropy, Linear Subspace

By model (3)

SigLIP
Dimensional collapse, Linear Subspace, Anisotropy, Linear Direction
SigLIP 2
Dimensional collapse, Linear Direction, Anisotropy, Linear Subspace
SigLIP ViT-B/16 · 203M
Anisotropy

Observations (7)

CLIP/SigLIP covariance splits into signal and shared-noise subspaces
Same concept, different SAE directions across modalities
Independent multimodal models share one orthogonal map across both modalities
Iso-energy SAE isolates a bimodal cross-modal subspace
The medical VLM modality gap is a tunable property whose reduction improves downstream performance, but full collapse is not optimal
Compositional generalization needs linear, additive, orthogonal per-concept factor vectors
VLM SAE concepts are single-modality yet gap-orthogonal

Papers

Your CLIP has 164 dimensions of noise: Exploring the embeddings covariance eigenspectrum of contrastively pretrained vision-language transformers (2026), Same Concept, Different Directions: Cross-Modal Feature Heterogeneity in Sparse Autoencoders (2026), Canonicalizing Multimodal Contrastive Representation Learning (2026), Cross-Modal Redundancy and the Geometry of Vision-Language Embeddings (2026), On the Cone Effect and Modality Gap in Medical Vision-Language Embeddings (2026), Compositional Generalization Requires Linear, Orthogonal Representations in Vision Embedding Models (2026), Interpreting the Linear Structure of Vision-Language Model Embedding Spaces (2025)