MMCR trains competitive SSL via a distinct low-dim, large-radius geometry

measured in 1 paper

Yerxa et al. derive a differentiable manifold-capacity objective (negative nuclear norm of augmentation centroids) and train ResNet-50 on ImageNet, reaching 69.5-72.1% linear-eval accuracy matching SimCLR/MoCo-v2/BYOL/SwAV [yerxa-etal-2023-maximum-manifold-capacity-representations] MFTMA analysis shows MMCR achieves this via a distinct geometry: lower manifold dimensionality but larger radius, emerging in late representational stages [yerxa-etal-2023-maximum-manifold-capacity-representations] MMCR yields the highest participation ratio (279.2) and a spectral decay coefficient (1.04) closest to the target of 1 among six SSL methods, and is competitive on macaque V2/V4 neural predictivity [yerxa-etal-2023-maximum-manifold-capacity-representations]

Structure

Linear Separability

Context

manifold capacity as a differentiable training objective, nuclear norm of the augmentation-centroid matrix, elliptical-manifold closed-form radius/dimensionality, participation ratio and spectral decay coefficient, self-supervised learning (contrastive vs. non-contrastive)

Confirmed in models

ResNet-50 (Maximum Manifold Capacity Representations, self-supervised, ImageNet) · ResNet, ResNet-50 (MoCo v2, unsupervised contrastive pretraining, ImageNet) · ResNet, ResNet-50 (SimCLR contrastive pretraining, ImageNet) · SimCLR

Method

Maximum Manifold Capacity Representations (MMCR) objective, Mean-Field Theoretic Manifold Analysis (MFTMA)

Papers

Learning Efficient Coding of Natural Images with Maximum Manifold Capacity Representations — Yerxa, Thomas, Kuang, Yilun, Simoncelli, Eero, Chung, SueYeon2023 · arXiv:2303.03307