MATH · IN · MODELS

CLIP's modality gap is a contrastive uniformity failure

measured in 1 paper

- The CLIP "modality gap" is reframed as a contrastive gap: a failure of uniformity in which embeddings occupy a narrow, low-dimensional cone instead of filling the hypersphere. [fahim-etal-2024-contrastive-gap] - Adding Wang-and-Isola uniformity and alignment terms plus a cross-modal uniformity term collapses the centroid gap 0.66 to 0.08 and lowers modality linear separability 1.00 to 0.73. [fahim-etal-2024-contrastive-gap] - The added terms improve multimodal arithmetic (SIMAT 36.02 to 42.18) and zero-shot classification while leaving retrieval roughly flat. [fahim-etal-2024-contrastive-gap] - Tested on CLIP ViT-B/32 fine-tuned on MS-COCO at reduced embedding dimensions (32/64/128D). [fahim-etal-2024-contrastive-gap]

Structure

Context

contrastive gap, uniformity, alignment, cross-modal uniformity, linear separability, PCA explained variance, zero-shot classification, multi-modal arithmetic

Papers

It's Not a Modality Gap: Characterizing and Addressing the Contrastive Gap — Fahim, Abrar, Murphy, Alex, Fyshe, Alona2024 · arXiv:2405.18570