DINOv3

Meta AI

Structures found in this family (5)

Linear Separability, Linear Subspace, Anisotropy, Linear Direction, Linear Centroids Hypothesis

By model (1)

DINOv3 ViT-B/16 · 86M

Linear Separability, Linear Subspace, Anisotropy, Linear Direction, Linear Centroids Hypothesis

Observations (5)

Patch-cosine Gram structure groups objects, strongest in self-supervised ViTs

Block-sparse featurizers recover concepts as two- to four-dimensional subspaces in DINOv3, one curve manifold in InceptionV1, and steerable blocks in SDXL

Dimension-level activation outliers, measured as the ratio gamma of mean norm to per-token variation, predict sparse-autoencoder feature death across 454 model-layer combinations spanning language, vision, protein and genomic models

Compositional generalization needs linear, additive, orthogonal per-concept factor vectors

Jacobian centroids (row-sums of a sub-component's local input-output Jacobian) form cleaner, more transferable linear feature directions than raw activations across four real-model settings: ResNet50/DTD, DINOv2-vs-DINOv3 SAE dictionaries, GPT2-Large circuit discovery, and Llama-3.1-8B truth probes

Papers

Human-like Object Grouping in Self-Supervised Vision Transformers (2026), Structuring Sparsity: Block-Sparse Featurizers Capture Visual Concept Manifolds (2026), On the Relationship Between Activation Outliers and Feature Death in Sparse Autoencoders (2026), Compositional Generalization Requires Linear, Orthogonal Representations in Vision Embedding Models (2026), The Linear Centroids Hypothesis: Features as Directions Learned by Local Experts (2026)