The medical VLM modality gap is a tunable property whose reduction improves downstream performance, but full collapse is not optimal

- Restrepo et al. (2026) introduce a lightweight, post-hoc mechanism that keeps a pretrained vision-language model's encoders frozen while continuously controlling the cross-modal separation ("cone effect" / modality gap) between image and text embeddings via a single hyperparameter, applied to CLIP (ViT-B/32), SigLIP (ViT-B/16), BioMedCLIP and MedSigLIP (Sec. 2.2): embeddings are shifted along the centroid gap vector Delta=mu_v-mu_t by lambda in [0,1] and renormalized (Eq. 5). The paper also measures the cone effect with the mean resultant length R (Table 1): image/text embeddings concentrate in "narrow angular regions of the hypersphere", with R generally higher on medical than on natural datasets, though not for every backbone-dataset pair (e.g. MedSigLIP on MIMIC-CXR, R=0.726 text and 0.735 image). [restrepo-etal-2026-cone-effect-modality-gap-medical-vlm] - On four medical and four natural datasets with a linear probe on concatenated image/text embeddings (5 seeds), "moderate closure of the modality gap consistently improves linear-probe AUC" (Sec. 3, Fig. 3); medical-domain backbones show smaller initial cross-modal separation than generalist ones, and the abstract states medical datasets show stronger sensitivity to gap modulation (the results text says generalist models benefit more on medical data). [restrepo-etal-2026-cone-effect-modality-gap-medical-vlm] - Full collapse (lambda to 1) is not uniformly best: "performance in some cases slightly degrades as lambda -> 1, particularly for medical-domain models on BRSET and mBRSET", which the authors read as loss of modality-specific information; the Conclusion also says closing the gap is "always desirable" relative to lambda=0 with slight dips for certain pairs. The authors frame the gap as tunable rather than something to be universally minimized. [restrepo-etal-2026-cone-effect-modality-gap-medical-vlm]