MATH · IN · MODELS

Vision Transformer (ViT)

Google Research

Structures found in this family (7)

Linear Separability, Linear Subspace, Polytope (Simplex), Aristotelian Representation Hypothesis, Intrinsic-dimension profile across depth, Platonic Representation Hypothesis, Linear Direction

By model (6)

MAE ViT-Base (Masked Autoencoder) · 86M
Linear Separability, Linear Subspace, Polytope (Simplex), Aristotelian Representation Hypothesis
ViT-Base · 86M
Linear Separability, Aristotelian Representation Hypothesis, Platonic Representation Hypothesis, Linear Subspace
ViT-Small · 22M
Aristotelian Representation Hypothesis, Platonic Representation Hypothesis
MAE ViT-Large (Masked Autoencoder) · 307M
Linear Subspace
ViT-B/16 (ImageNet-21k, supervised) · 86M
Linear Direction
Vision Transformer (ViT, image classifier, various sizes, ImageNet)
Intrinsic-dimension profile across depth

Observations (12)

Patch-cosine Gram structure groups objects, strongest in self-supervised ViTs
DINOv2 and Stable Diffusion encode depth/normals via nonlinear dense probes
SSL encoders collapse variance along each task's decision axis, aiding few-shot transfer
After correcting CKA and mutual k-NN alignment for a proven width/depth-dependent null-baseline confound (O(d/n) for global spectral metrics vs O(k/n) for local-neighborhood metrics), the Platonic Representation Hypothesis's global cross-modal convergence largely vanishes while its local-neighborhood convergence survives, across real image-text and video-language pretrained model pairs
A shared PCA basis's leading components separate figure from ground
Lower local intrinsic dimension predicts vision-model alignment and generalization
MAE per-class patch-token subspaces rotate apart with depth
Mid-level vision decodability weakly correlates with recognition accuracy
Re-expressing embeddings as cosine similarity to a fixed anchor set yields a rotation/reflection/rescaling-invariant coordinate system in which decoders and classifiers trained on one independently-trained encoder can be stitched, with zero fine-tuning, onto a completely different independently-trained encoder's representations of the same domain
A single linear ViT direction encodes depth and causally drives depth estimation
SVD-extracted ViT-L task subspaces recover most full-feature performance
ICA after PCA-whitening recovers a universal sparse axis-aligned decomposition

Papers

Human-like Object Grouping in Self-Supervised Vision Transformers (2026), Probing the 3D Awareness of Visual Foundation Models (2024), Directional Neural Collapse Explains Few-Shot Transfer in Self-Supervised Learning (2026), Revisiting the Platonic Representation Hypothesis: An Aristotelian View (2026), From Local Cues to Global Percepts: Emergent Gestalt Organization in Self-Supervised Vision Models (2025), Local Intrinsic Dimension of Representations Predicts Alignment and Generalization in AI Models and the Human Brain (2026), Robust Representation Learning in Masked Autoencoders (2026), Probing the Mid-level Vision Capabilities of Self-Supervised Learning (2024), Relative Representations Enable Zero-Shot Latent Space Communication (2022), From Edges to Depth: Probing the Spatial Hierarchy in Vision Transformers (2026), Understanding Geometric Representations in Self-Supervised Vision Transformers via Subspace Intervention (2026), Discovering Universal Geometry in Embeddings with ICA (2023)