Pretrained ViTs show block-wise layer-layer cosine-similarity phases, and DINOv2 depth trajectories converge to class-dependent angular basins with low-rank late updates

- Jacobs, Fel, Hakim, Brondetta, Ba and Keller compute token-wise layer-layer cosine-similarity matrices across diverse pretrained ViTs and find contiguous blocks of high mutual similarity separated by lower-similarity boundaries; a max-cut partition on DINOv2 ViT-B gives a few phases. [jacobs-etal-2025-block-recurrent-dynamics-in-vision-transformers] - Weight-tied block-recurrent surrogates (Raptor) with 2 blocks recover about 96 percent of DINOv2 ViT-B ImageNet-1k linear-probe accuracy, supporting that the phases reflect reusable computation rather than similarity alone. [jacobs-etal-2025-block-recurrent-dynamics-in-vision-transformers] - Depth trajectories on the unit sphere converge directionally (S-shaped cosine-to-final curves) into compact class-dependent basins in a PCA subspace and bend back after small perturbations; stable and effective rank of layer updates fall to about six in the final phase. Described by the authors as consistent with low-dimensional angular attractors, not a proven attractor. [jacobs-etal-2025-block-recurrent-dynamics-in-vision-transformers] - The foundation-model experiments use DINOv2 ViT-B with registers (Appendix A: "trained on top of DINOv2 (ViT-B with registers)"), hence the dinov2-base-registers tag; the cross-model similarity matrices (Figure 1) are in an image whose model list is not in the text and is not tagged, and the stochastic-depth and recurrence-emergence experiments use small ViT-B/14 models the authors trained on CIFAR-100. Dynamical analyses (angular basins, rank collapse) are shown for the DINOv2 backbone (1,000 images from 5 ImageNet classes for the PCA view). [jacobs-etal-2025-block-recurrent-dynamics-in-vision-transformers]