Singular Vector Canonical Correlation Analysis (SVCCA)

methods / Representation Alignment / Singular Vector Canonical Correlation Analysis (SVCCA)

Singular Vector Canonical Correlation Analysis (SVCCA)

Techniqueintermediate

SVD-truncates each of two representations to its top variance-retaining directions, then applies Canonical Correlation Analysis (CCA) to linearly align the two truncated spaces, reporting the mean canonical correlation as a single similarity score — the historical predecessor CKA was designed to improve on.

Used in (5 observations)

A 141-hypothesis automated topology/geometry audit of scGPT and Geneformer gene-embedding representations finds statistically significant persistent homology in nearly every transformer layer and a canonical-correlation cross-model alignment of 0.80 between the two independently-trained modelsmeasured

structure: Persistent-homology / Betti profile across depth, Platonic Representation Hypothesis · models: scGPT (whole-human pretrained checkpoint), Geneformer · paper: What Topological and Geometric Structure Do Biological Foundation Models Learn? Evidence from 141 Hypotheses

Independently-trained embedding models share linear structure (CKA 0.60-0.88)measured

structure: Linear Subspace · models: text-embedding-3-small, Cohere Embed, Gemini embedding-001, Qwen3-Embedding-8B, E5-mistral-7b-instruct, Llama-3-8B, Qwen2.5-7B-Instruct, Qwen2.5-14B-Instruct, Llama-3.2-1B, OLMo-7B · paper: Characterizing Linear Alignment Across Language Models

SAE feature spaces (not raw activations) trained independently on Pythia-70m/160m, Gemma-1/2, and Llama-3/3.1 show high SVCCA and RSA similarity at matched middle layers after activation-correlation feature pairing, far above random-pairing and randomized-weight-model baselines, and semantically matched concept subspaces (e.g. Emotions) are more similar across models than the feature space as a wholemeasured

structure: Platonic Representation Hypothesis · models: Pythia-70M, Pythia-160M, Gemma-2B, Gemma-2-2B, Gemma-2-9B, Llama-3-8B-Instruct, Llama-3.1-8B · paper: Quantifying Feature Space Universality Across Large Language Models via Sparse Autoencoders, Semantic Convergence: Investigating Shared Representations Across Scaled LLMs

Multilingual NMT encoders converge across languages with depthmeasured

structure: Platonic Representation Hypothesis, Linear Subspace · models: Google Multilingual NMT (Transformer-Big, 103 languages) · paper: Investigating Multilingual NMT Representations at Scale

Number-task subspaces are distinct yet linearly inter-transformablemeasured

structure: Linear Subspace, Linear Direction · models: BERT-base-uncased, GPT-2 Small, Qwen-2.5-7B, Qwen2.5-Math-7B · paper: The Representational Geometry of Number