MATH · IN · MODELS

BYOL

DeepMind

Structures found in this family (1)

Dimensional collapse

By model (2)

BYOL (ResNet-18 backbone, trained from scratch on CIFAR-10 with solo-learn)
Dimensional collapse
BYOL (ResNet-50 backbone)
no structures recorded for this checkpoint specifically

Observations (1)

In BYOL, SimSiam, SwAV and DINO on CIFAR-10 the target branch has higher effective rank than the online branch, which the authors prove follows from a low-pass online spectral filter and argue alleviates dimensional collapse

Papers

Towards a Unified Theoretical Understanding of Non-contrastive Learning via Rank Differential Mechanism (2023)