MAE per-class patch-token subspaces rotate apart with depth

Shrivastava et al. take the pretrained MAE ViT-Base encoder and, per class, SVD the visible patch-token embeddings at each layer to get a k-dimensional class subspace [shrivastava-etal-2026-robust-representation-learning-mae] The smallest principal angle between per-class subspaces grows with depth, so class-specific subspaces rotate apart and become well-separated without labels, corroborated by t-SNE around layers 7-9 [shrivastava-etal-2026-robust-representation-learning-mae] This class structure is linked to MAE's persistent global attention (mean attention distance ~80-120px at all layers, unlike standard ViT's local-to-global progression) [shrivastava-etal-2026-robust-representation-learning-mae] Fine-tuned MAE stays above 80% accuracy under Gaussian blur and above 60% at 90% occlusion, with correctly-classified images showing higher clean-vs-perturbed cosine similarity; no causal intervention is performed [shrivastava-etal-2026-robust-representation-learning-mae]