MATH · IN · MODELS

CLIP (Contrastive Language-Image Pretraining)

OpenAI

Structures found in this family (11)

Linear Direction, Linear Separability, Anisotropy, Dimensional collapse, Linear Subspace, Polytope (Simplex), Aristotelian Representation Hypothesis, Platonic Representation Hypothesis, Spatial-frequency (spectral) accessibility profile across depth, Intrinsic-dimension profile across depth, Cone

Hypotheses argued for by this family (1)

Platonic Representation Hypothesis

By model (9)

CLIP ViT-B/32
Linear Direction, Linear Separability, Anisotropy, Dimensional collapse, Polytope (Simplex), Platonic Representation Hypothesis, Linear Subspace, Intrinsic-dimension profile across depth, Cone
CLIP ViT-L/14
Linear Separability, Dimensional collapse, Anisotropy, Linear Direction, Linear Subspace, Aristotelian Representation Hypothesis, Spatial-frequency (spectral) accessibility profile across depth
CLIP ViT-B/16 · 86M
Linear Separability, Anisotropy, Linear Direction, Dimensional collapse, Aristotelian Representation Hypothesis
CLIP ResNet-50
Linear Subspace, Linear Direction, Anisotropy
CLIP
Linear Direction, Anisotropy
CLIP ResNet-50x16
Linear Subspace
CLIP ViT-bigG/14 (Kandinsky 2.2 image encoder)
Linear Direction
CLIP ViT-H/14
Linear Separability
OpenCLIP ViT-B/16 (LAION-2B)
Linear Direction

Observations (31)

A value-like success direction is recoverable from frozen VLAs
CLIP binding is linearly decodable within a modality but collapses across
CLIP's modality gap is a provable, tunable cone effect
CLIP/SigLIP covariance splits into signal and shared-noise subspaces
CLIP embeddings lie on two tilted offset ellipsoids
CLIP's modality gap is a contrastive uniformity failure
Same concept, different SAE directions across modalities
DINOv2 and Stable Diffusion encode depth/normals via nonlinear dense probes
SSL encoders collapse variance along each task's decision axis, aiding few-shot transfer
After correcting CKA and mutual k-NN alignment for a proven width/depth-dependent null-baseline confound (O(d/n) for global spectral metrics vs O(k/n) for local-neighborhood metrics), the Platonic Representation Hypothesis's global cross-modal convergence largely vanishes while its local-neighborhood convergence survives, across real image-text and video-language pretrained model pairs
Independently pretrained vision (DINOv2) and text (MiniLM) encoders develop graph-Laplacian eigenvalue spectra of near-identical intrinsic complexity, yet the functional map between their spectral eigenbases is far from diagonal or orthogonal -- a decoupling of 'how much structure' from 'how it is oriented' that bounds spectral cross-modal alignment methods and underperforms ambient-space alignment on retrieval
Independent multimodal models share one orthogonal map across both modalities
Iso-energy SAE isolates a bimodal cross-modal subspace
vec2vec learns a fully unpaired translation between six independently-trained text embedding models (gtr, clip, e5, gte, stella, granite), recovering cosine similarity up to 0.92 and top-1 retrieval accuracy up to 100% with no paired training data, and an ablation shows the explicit Gram-matrix-preservation (VSP) loss is causally necessary -- removing it alone collapses top-1 accuracy from 0.91 to 0.00
Fewer than 25 register neurons with consistent directions cause ViT outlier tokens
A non-monotonic spatial-frequency accessibility profile across depth in real CLIP and DINOv2 vision transformers
Representational similarity analysis shows 15 language and vision-language models align more strongly with brain regions whose activity is consistent across sentence, word-cloud, and image presentations of the same concept than with less modality-consistent regions
A shared PCA basis's leading components separate figure from ground
One unconstrained linear map injects vision embeddings into frozen GPT-J
An affine map between two vision models matches their PCA components
MST intrinsic dimension of frozen SSL features predicts probe accuracy
Fitted part-of-speech CLIP subspaces disentangle content from appearance
SAE directions show CLIP encodes abstract concepts, DINOv2 fragmented visual features
Sparse nonnegative concept decomposition enables causal CLIP editing
A causally-validated Diffusion Steering Lens tracks real head-ablation effects in a ViT
Modality gap converges to angle between collapsed hyperplanes
Low-temperature InfoNCE is the modality gap's cause
Compositional generalization needs linear, additive, orthogonal per-concept factor vectors
Multi-object scene embeddings decompose hierarchically as sums of object embeddings
VLM embeddings show an emergent radial entailment hierarchy, improvable by fine-tuning
VLM SAE concepts are single-modality yet gap-orthogonal

Papers

What Frozen VLAs Already Know About Success: A Probing Study of Value-Like Structure in Foundation Robot Policies (2026), CLIP Behaves Like a Bag-of-Words Model Cross-modally but Not Uni-modally (2026), Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning (2022), Your CLIP has 164 dimensions of noise: Exploring the embeddings covariance eigenspectrum of contrastively pretrained vision-language transformers (2026), The Double-Ellipsoid Geometry of CLIP (2025), It's Not a Modality Gap: Characterizing and Addressing the Contrastive Gap (2024), Same Concept, Different Directions: Cross-Modal Feature Heterogeneity in Sparse Autoencoders (2026), Probing the 3D Awareness of Visual Foundation Models (2024), Directional Neural Collapse Explains Few-Shot Transfer in Self-Supervised Learning (2026), Revisiting the Platonic Representation Hypothesis: An Aristotelian View (2026), On the Spectral Geometry of Cross-Modal Representations: A Functional Map Diagnostic for Multimodal Alignment (2026), Canonicalizing Multimodal Contrastive Representation Learning (2026), Cross-Modal Redundancy and the Geometry of Vision-Language Embeddings (2026), Harnessing the Universal Geometry of Embeddings (2026), Vision Transformers Don't Need Trained Registers (2025), Beyond Compression: Quantifying Spectral Accessibility in Vision Representations (2026), Language models align with brain regions that represent concepts across modalities (2025), From Local Cues to Global Percepts: Emergent Gestalt Organization in Self-Supervised Vision Models (2025), Linearly Mapping from Image to Text Space (2023), Text-to-Concept (and Back) via Cross-Model Alignment (2023), IdEst: Assessing Self-Supervised Learning Representations via Intrinsic Dimension (2026), Parts of Speech-Grounded Subspaces in Vision-Language Models (2023), Interpretable and Testable Vision Features via Sparse Autoencoders (2025), Interpreting CLIP with Sparse Linear Concept Embeddings (SpLiCE) (2024), Decoding Vision Transformers: The Diffusion Steering Lens (2025), Decipher the Modality Gap in Multimodal Contrastive Learning: From Convergent Representations to Pairwise Alignment (2025), On the Modality Gap and the Contrastive Loss in Multi-modal Representation Learning (2026), Compositional Generalization Requires Linear, Orthogonal Representations in Vision Embedding Models (2026), How Can Embedding Models Bind Concepts? (2026), Emergent Visual-Semantic Hierarchies in Image-Text Representations (2024), Interpreting the Linear Structure of Vision-Language Model Embedding Spaces (2025)