Block-sparse featurizers recover concepts as two- to four-dimensional subspaces in DINOv3, one curve manifold in InceptionV1, and steerable blocks in SDXL
measured in 1 paper- Fel et al. train block-sparse featurizers (BSFs) on frozen vision models. On DINOv3 (final-layer ViT-B patch activations), the mean stable rank of blocks saturates near 3 even when blocks are allotted up to 16 dimensions, i.e. concepts are on average about two- to four-dimensional (Fig 8), and a minimum-description-length comparison favors blocks of dimension 2-4 over single-direction SAEs. In InceptionV1, one b=16 block carries the whole curve-orientation manifold, with the first three Fourier harmonics explaining 89% of block variance (59/18/12%); this is not a 2-4D claim. In SDXL (layer down.2.1), steering within a block (e.g. a pretzel concept) produces semantically related images, shown qualitatively. The paper describes block activations as tracing curved low-dimensional manifolds, so the linear-subspace node is tentative. [fel-etal-2026-block-sparse-featurizers-concepts-as-low-dimensional-manifolds]