ICA axis decomposition

methods / Dictionary Learning / ICA axis decomposition

Techniqueadvanced

Whitens (PCA-decorrelates) an embedding matrix, then applies Independent Component Analysis to find a privileged, non-orthogonal rotation of the whitened axes under which individual coordinates are maximally statistically independent and sparse — recovering interpretable, axis-aligned semantic components that PCA's rotation-free variance-maximizing axes do not.

Used in (1 observation)

ICA after PCA-whitening recovers a universal sparse axis-aligned decompositionmeasured

structure: Linear Subspace · models: word2vec SGNS (text8, 300d), FastText (bag-of-word-vectors), BERT-base-uncased, ViT-Base, ResNet-18 (supervised, ImageNet) · paper: Discovering Universal Geometry in Embeddings with ICA