CLIP/SigLIP covariance splits into signal and shared-noise subspaces

measured in 1 paper

- The embedding covariance eigenspectrum of six CLIP/SigLIP models shows a sharp elbow (threshold ~10^-3.6) separating a semantic signal subspace from a shared noise subspace. [grzywaczewski-etal-2026-clip-covariance-eigenspectrum-noise] - The noise subspace's dimensionality is quantified per model (CLIP ViT-L/14 164/768 = 21%; SigLIP2 ViT-L/16 405/1024 = 40%) and is shared across unrelated classes (>50% overlap, >90% for the largest models). [grzywaczewski-etal-2026-clip-covariance-eigenspectrum-noise] - Projecting the noise subspace away leaves ImageNet zero-shot accuracy essentially unchanged (CLIP ViT-L/14 91.2% to 91.1%) while removing the same number of random directions degrades it (to 89.3%); text-image cosine similarity even improves. [grzywaczewski-etal-2026-clip-covariance-eigenspectrum-noise] - Tested on CLIP ViT-B/16, ViT-B/32, ViT-L/14, SigLIP ViT-B/16, SigLIP2 ViT-B/16 and ViT-L/16. [grzywaczewski-etal-2026-clip-covariance-eigenspectrum-noise]

Structure

Dimensional collapse

Context

covariance-eigenspectrum elbow separates a semantic signal subspace from a noise subspace, noise-subspace dimensionality is precisely quantified and varies by model (3%-40% of ambient dims), the noise subspace is shared across unrelated classes (over 90% overlap in the largest models), ablating the noise subspace preserves accuracy while ablating an equal-size random subspace degrades it

Confirmed in models

CLIP ViT-B/16 · CLIP (Contrastive Language-Image Pretraining), CLIP ViT-B/32 · CLIP (Contrastive Language-Image Pretraining), CLIP ViT-L/14 · CLIP (Contrastive Language-Image Pretraining), SigLIP · SigLIP, SigLIP 2 · SigLIP

Method

PCA, Causal interventions (steering)

Papers

Your CLIP has 164 dimensions of noise: Exploring the embeddings covariance eigenspectrum of contrastively pretrained vision-language transformers — Grzywaczewski, Jakub, Płudowski, Dawid, Biecek, Przemysław2026 · arXiv:2605.14893