ICA after PCA-whitening recovers a universal sparse axis-aligned decomposition

measured in 1 paper

Yamagiwa et al. apply PCA-whitening then ICA to their own text8-trained word2vec (SGNS), cross-lingual fastText (7 languages), BERT contextual embeddings, and image encoders (ViT, ResMLP, Swin, ResNet, RegNet) [yamagiwa-etal-2023-discovering-universal-geometry-ica] The independent axes are individually interpretable and sparse, unlike PCA arbitrary rotation [yamagiwa-etal-2023-discovering-universal-geometry-ica] Specific axes can be matched across languages, models and modalities via cross-model component correlation, a privileged non-arbitrary decomposition [yamagiwa-etal-2023-discovering-universal-geometry-ica] No causal intervention is performed; axis interpretability is partly qualitative and universality depends on whitening choices [yamagiwa-etal-2023-discovering-universal-geometry-ica]

Structure

Linear Subspace

Context

ICA axis decomposition, universal semantic geometry, cross-model/cross-lingual/cross-modal component matching

Confirmed in models

word2vec SGNS (text8, 300d) · word2vec, FastText (bag-of-word-vectors) · FastText, BERT-base-uncased · BERT, ViT-Base · Vision Transformer (ViT), ResNet-18 (supervised, ImageNet) · ResNet

Method

ICA axis decomposition

Papers

Discovering Universal Geometry in Embeddings with ICA — Yamagiwa, Hiroaki, Oyama, Momose, Shimodaira, Hidetoshi2023 · arXiv:2305.13175