Contrastive SSL embeddings undergo provable covariance-spectrum rank collapse
measured in 1 paper- Contrastive self-supervised embeddings (SimCLR, ResNet-50, ImageNet) undergo dimensional collapse: many singular values of the embedding covariance drop to numerical zero, in both the projection head and the backbone. [jing-etal-2022-dimensional-collapse] - Two causes are proven for a linear model under InfoNCE: strong augmentation drives weights toward zero wherever augmentation variance exceeds data variance (Lemma 1), and multilayer implicit regularization under SGD drives low rank even under weak augmentation (Corollary 2). [jing-etal-2022-dimensional-collapse] - The collapse is diagnosed via the log singular-value spectrum of the covariance; DirectCLR (a fixed low-rank projector) recovers accuracy (ImageNet linear-probe Top-1 66.5% full projector, 51.5% none, 62.7% DirectCLR). [jing-etal-2022-dimensional-collapse]