MATH · IN · MODELS

CLIP's modality gap is a provable, tunable cone effect

measured in 1 paper

- Each modality's embeddings collapse into a narrow cone present even at random initialization; two distinct cones plus low temperature (tau=1/100) leave a persistent image-text modality gap. [liang-etal-2022-modality-gap] - The default gap magnitude ||Delta||=0.82 is the contrastive-loss global minimum, so the gap is a geometric consequence of the cone effect rather than an artifact. [liang-etal-2022-modality-gap] - Causally shifting embeddings along the gap direction changes downstream behavior: widening the gap 0.82 to 0.97 reduces FairFace denigration, with small zero-shot shifts (CIFAR-10 0.9013 to 0.9081). [liang-etal-2022-modality-gap] - Zero-shot accuracy experiments use CLIP ViT-B/16, fairness uses ViT-B/32; the paper also documents the gap in VideoCLIP, ConVIRT and CLASP. [liang-etal-2022-modality-gap]

Context

modality gap, narrow cone, cosine similarity, contrastive temperature, embedding shift intervention, zero-shot accuracy, fairness/bias tradeoff

Papers

Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning — Liang, Weixin, Zhang, Yuhui, Kwon, Yongchan, Yeung, Serena, Zou, James2022 · arXiv:2203.02053