OpenCLIP

LAION

Structures found in this family (4)

Linear Direction, Anisotropy, Linear Subspace, Cone

By model (3)

OpenCLIP ViT-B/32

Linear Direction, Anisotropy, Linear Subspace

OpenCLIP ViT-L/14

Linear Subspace, Anisotropy, Linear Direction

OpenCLIP ViT-H/14

Cone

Observations (5)

Same concept, different SAE directions across modalities

Iso-energy SAE isolates a bimodal cross-modal subspace

Sparse nonnegative concept decomposition enables causal CLIP editing

Compositional generalization needs linear, additive, orthogonal per-concept factor vectors

VLM embeddings show an emergent radial entailment hierarchy, improvable by fine-tuning

Papers

Same Concept, Different Directions: Cross-Modal Feature Heterogeneity in Sparse Autoencoders (2026), Cross-Modal Redundancy and the Geometry of Vision-Language Embeddings (2026), Interpreting CLIP with Sparse Linear Concept Embeddings (SpLiCE) (2024), Compositional Generalization Requires Linear, Orthogonal Representations in Vision Embedding Models (2026), Emergent Visual-Semantic Hierarchies in Image-Text Representations (2024)