An affine map between two vision models matches their PCA components

measured in 1 paper

Moayeri et al. fit an affine least-squares map between the activation spaces of independently-trained vision models (supervised/robust ResNets, Swin/DeiT/ConViT, self-supervised MoCo/DINO-ViT-S/SimCLR ResNets and ViTs, and CLIP), scored by R^2 [moayeri-etal-2023-text-to-concept-and-back-via-cross-model-alignment] The fitted map explains R^2 above 0.6, and the top principal components of the two aligned spaces correspond approximately one-to-one [moayeri-etal-2023-text-to-concept-and-back-via-cross-model-alignment] Aligning a vision encoder into CLIP's concept space enables zero-shot concept-bottleneck classification at up to 93.8% accuracy and over 92% concept-to-text relevance [moayeri-etal-2023-text-to-concept-and-back-via-cross-model-alignment] The analysis is passive, with no causal intervention on either model [moayeri-etal-2023-text-to-concept-and-back-via-cross-model-alignment]

Structure

Linear Subspace

Context

fitting a linear/affine map between two independently-trained models' full activation spaces (rather than one extracted direction), then checking whether their top-PCA-component structure also corresponds, a passive (non-causal) cross-model alignment measurement, distinct from directional steering/ablation

Confirmed in models

CLIP ResNet-50 · CLIP (Contrastive Language-Image Pretraining), CLIP ViT-B/32 · CLIP (Contrastive Language-Image Pretraining), DINO ViT-S/16 · DINO, ResNet-50 (SimCLR contrastive pretraining, ImageNet) · SimCLR, ResNet-50 (MoCo v2, unsupervised contrastive pretraining, ImageNet) · ResNet

Method

Learned linear/affine map between two representation spaces

Papers

Text-to-Concept (and Back) via Cross-Model Alignment — Moayeri, Mazda, Rezaei, Keivan, Sanjabi, Maziar, Feizi, Soheil2023 · arXiv:2305.06386