Audio-text embeddings split into shared head and private tail

measured in 1 paper

- A cross-modal PLS-SVD decomposition splits CLAP audio-text embeddings into a static mean component (the gap), a compact shared semantic head (~100 of 1024 dims carrying most cross-modal variance), and a diffuse modality-private tail. [zhu-etal-2026-comet-audio-modality-gap] - Truncating to the top-100 shared axes (PLSHead) preserves retrieval (Clotho Text->Audio R@1 17.42 to 17.32) while cutting MeanR 42.36 to 36.30. [zhu-etal-2026-comet-audio-modality-gap] - A PCA-head ablation collapses retrieval to R@1 ~0.06/0.00, isolating the PLS shared axes as causally responsible; removing the tail drops captioning BLEU-4 to 5.9-7.1. [zhu-etal-2026-comet-audio-modality-gap] - Tested on CLAP HTSAT-BERT (zero-shot) and the DRCap CLAP backbone. [zhu-etal-2026-comet-audio-modality-gap]

Structure

Linear Subspace, Anisotropy

Context

cross-modal PLS-SVD decomposition, shared semantic head, modality-private tail, PLSHead truncation, Projection Decoding reinterpretation, comet structure

Confirmed in models

CLAP (HTSAT-BERT-ZS, trained on WavCaps) · CLAP (Contrastive Language-Audio Pretraining), DRCap's CLAP (trained on WavCaps + SoundVECaps) · CLAP (Contrastive Language-Audio Pretraining)

Method

Cross-Modal PLS-SVD Decomposition, Geometric analysis, Causal interventions (steering)

Papers

COMET: Concept Space Dissection of the Modality Gap in Audio-Text Multimodal Contrastive Embeddings — Zhu, Yonggang, Gao, Liting, Men, Aidong, Wang, Wenwu2026 · arXiv:2605.29628