In BYOL, SimSiam, SwAV and DINO on CIFAR-10 the target branch has higher effective rank than the online branch, which the authors prove follows from a low-pass online spectral filter and argue alleviates dimensional collapse

- Zhuo et al. (2023) measure the effective rank (erank) of the normalized outputs of the online and target branches for BYOL, SimSiam, SwAV and DINO (ResNet-18 trained with solo-learn on CIFAR-10): the target branch consistently has higher rank than the online branch and the online rank rises during training after warmup (Fig. 1); they find a high degree of eigenspace alignment between the online and target correlation matrices (always above 0.7 for DINO; Fig. 3), and the filter g = sqrt(lambda^p / lambda^z) is roughly monotonically increasing for all four methods, i.e. a low-pass online filter (Fig. 4b). [zhuo-etal-2023-rank-differential-mechanism] - Under their Hypothesis 1 (eigenspace alignment), Theorem 3 proves that a monotonically increasing g (low-pass online filter) gives erank(C_p) < erank(C_z), and Theorem 4 shows gradient descent then raises the online effective rank; a fully symmetric variant (SymSimSiam) avoids complete collapse but suffers more dimensional collapse than SimSiam (Fig. 2c) and has lower linear-probe accuracy (Fig. 2a), which the authors take to show that asymmetry is what alleviates dimensional collapse. [zhuo-etal-2023-rank-differential-mechanism] - New asymmetric variants built from the theory (fixed online low-pass filters g(sigma) = sigma, log sigma, log(1 + sigma), log(1 + sigma^2), and target high-pass filters h(sigma) = sigma^p with -1 <= p < 0) are compared with SimSiam (and, in Appendix C, in the BYOL framework): on CIFAR-10/100 and ImageNet-100 target high-pass filters often outperform SimSiam in linear-probe accuracy at 100 epochs (e.g. 78.17 vs 74.00 on CIFAR-10) while online filters are comparable (Tables 1-2); on ImageNet-1k, SimSiam with a ResNet-50 backbone scores 67.97 (learnable predictor) vs 64.67 (log(1 + sigma) online filter) and 67.73 (sigma^-0.3 target filter) top-1 (Table 3). [zhuo-etal-2023-rank-differential-mechanism]