MATH · IN · MODELS
methods / Direction Extraction / Additive per-concept factor recovery and orthogonality scoring

Additive per-concept factor recovery and orthogonality scoring

Techniqueintermediate

Recovers a per-concept difference vector for each factor of variation from paired embeddings (e.g. same scene differing only in one attribute), then scores how well an embedding reconstructs as a sum of these factor vectors ($R^2$) and how mutually orthogonal the recovered factors are — turning "is this representation compositional" into two separate quantified numbers.

Used in (2 observations)

Compositional generalization needs linear, additive, orthogonal per-concept factor vectorsmeasured
structure: Linear Direction · models: CLIP ViT-L/14, OpenCLIP ViT-L/14, SigLIP, SigLIP 2, DINOv2 ViT-B/14, DINOv3 ViT-B/16 · paper: Compositional Generalization Requires Linear, Orthogonal Representations in Vision Embedding Models
Multi-object scene embeddings decompose hierarchically as sums of object embeddingsmeasured
structure: Linear Direction · models: CLIP ViT-L/14, DINOv2 ViT-B/14 · paper: How Can Embedding Models Bind Concepts?