MATH · IN · MODELS

DINO

Meta AI

Structures found in this family (4)

Linear Separability, Intrinsic-dimension profile across depth, Linear Subspace, Dimensional collapse

By model (6)

DINO ViT-B/16 · 86M
Linear Separability, Linear Subspace
ViT-Small (12 layers, 6 heads, dim 384) trained with DINO from scratch on CIFAR-100, 100 epochs · 21M
Intrinsic-dimension profile across depth
ViT-Small (12 layers, 6 heads, dim 384) trained with DINO from scratch on Tiny ImageNet, 100 epochs · 21M
Intrinsic-dimension profile across depth
ViT-Small (12 layers, 6 heads, dim 384) trained with DINO from scratch on UC Merced, 100 epochs · 21M
Intrinsic-dimension profile across depth
DINO ViT-S/16 · 21M
Linear Subspace
DINO (ResNet-18 backbone, trained from scratch on CIFAR-10 with solo-learn)
Dimensional collapse

Observations (5)

Patch-cosine Gram structure groups objects, strongest in self-supervised ViTs
DINO ViT-Small trained from scratch shows a U-shaped layer-wise effective-dimension profile on object-centric data and stays high-rank on texture data
Mid-level vision decodability weakly correlates with recognition accuracy
An affine map between two vision models matches their PCA components
In BYOL, SimSiam, SwAV and DINO on CIFAR-10 the target branch has higher effective rank than the online branch, which the authors prove follows from a low-pass online spectral filter and argue alleviates dimensional collapse

Papers

Human-like Object Grouping in Self-Supervised Vision Transformers (2026), The Inductive Bottleneck: Data-Driven Emergence of Representational Sparsity in Vision Transformers (2025), Probing the Mid-level Vision Capabilities of Self-Supervised Learning (2024), Text-to-Concept (and Back) via Cross-Model Alignment (2023), Towards a Unified Theoretical Understanding of Non-contrastive Learning via Rank Differential Mechanism (2023)