DINO ViT-Small trained from scratch shows a U-shaped layer-wise effective-dimension profile on object-centric data and stays high-rank on texture data

measured in 1 paper

Awadhiya trains ViT-Small (12 layers, D=384) from scratch with DINO for 100 epochs on UC Merced, Tiny ImageNet and CIFAR-100 and measures the layer-wise Effective Encoding Dimension (exponential of the Shannon entropy of the token-embedding covariance eigenvalue spectrum) [awadhiya-2025-the-inductive-bottleneck] Minimum EED in layers L2-L4 is about 23.0 percent of D for CIFAR-100, 30.5 percent for Tiny ImageNet and about 95.0 percent for UC Merced; Tiny ImageNet recovers to 92.5 percent EED at layer 11 [awadhiya-2025-the-inductive-bottleneck] The UC Merced profile stays above 90 percent rank throughout, so the compress-then-expand bottleneck depends on the dataset rather than being fixed by the architecture; the paper reports one ViT-Small per dataset and no causal intervention [awadhiya-2025-the-inductive-bottleneck]

Structure

Intrinsic-dimension profile across depth

Context

effective encoding dimension (spectral-entropy effective rank), inductive bottleneck, data-dependent compression

Confirmed in models

ViT-Small (12 layers, 6 heads, dim 384) trained with DINO from scratch on UC Merced, 100 epochs · DINO, ViT-Small (12 layers, 6 heads, dim 384) trained with DINO from scratch on Tiny ImageNet, 100 epochs · DINO, ViT-Small (12 layers, 6 heads, dim 384) trained with DINO from scratch on CIFAR-100, 100 epochs · DINO

Method

PCA

Papers

The Inductive Bottleneck: Data-Driven Emergence of Representational Sparsity in Vision Transformers — Awadhiya, Kanishk2025 · arXiv:2512.07331