MATH · IN · MODELS
methods / Representation Alignment

Representation Alignment

Category

Compare two representations' overall geometry to each other directly — do two models, modalities, or checkpoints measure similarity between the same datapoints in the same way — rather than analyzing the geometry of a single representation in isolation.

13 methods in this category

Representational Similarity Analysis (RSA)
Compares two representations of the same items by building one pairwise-distance matrix per representation and correlating the two matrices (e.g. via Kendall's tau), rather than comparing raw coordinates directly — the classical, originally-neuroscientific alignment technique this map's other representation-alignment methods refine.
15 papers
Orthogonal Procrustes alignment
Fits the orthogonal matrix $W^\star = \arg\min_{W\in O(d)} \|WX-Y\|_F = UV^\top$ (via the SVD of $YX^\top$) that best maps one representation onto another, then evaluates the fit functionally — e.g. via nearest-neighbor retrieval or downstream transfer — rather than only reporting a single aggregate similarity number.
11 papers
Centered Kernel Alignment (CKA)
Continuous statistic comparing two representations' kernel (pairwise-similarity) matrices via a normalized Frobenius inner product between them — the standard quantitative alternative to rank-based mutual-NN alignment for measuring how similar two representations' relative geometry is.
8 papers
Singular Vector Canonical Correlation Analysis (SVCCA)
SVD-truncates each of two representations to its top variance-retaining directions, then applies Canonical Correlation Analysis (CCA) to linearly align the two truncated spaces, reporting the mean canonical correlation as a single similarity score — the historical predecessor CKA was designed to improve on.
6 papers
Generalized Procrustes Analysis (GPA)
Finds the optimal orthogonal transformation (rotation/reflection, no scaling or shearing) aligning one configuration of points onto another, then reports 1 minus the residual misalignment as a direct shape-similarity score — unlike RSA, this compares raw coordinate configurations rather than pairwise-distance matrices, so it is sensitive to genuine shape (not just relative distances).
2 papers
Learned linear/affine map between two representation spaces
Fits a linear or affine map (via least squares / SGD, with no orthogonality constraint) from one model's activation space onto another's, scored by R^2 or by downstream task performance — a general-linear alternative to orthogonal Procrustes for cross-model alignment and cross-modal transfer.
2 papers
Mutual nearest-neighbor alignment
Measures agreement between two representations' kernels as the mean overlap of their k-nearest-neighbor sets around each datapoint, normalized by k — a rank-based alternative to continuous kernel-alignment statistics like CKA.
2 papers
Adversarial unpaired representation translation (vec2vec)
Learns a map between two embedding spaces with no paired correspondence at all — via GAN-style adversarial training on both outputs and latents plus a Vector Space Preservation (VSP) loss that explicitly matches pairwise inner products (Gram matrices) between input and translated embeddings — turning cross-model geometric convergence into a constructive, unsupervised translation.
1 paper
CKNNA representational alignment
Centered Kernel Nearest-Neighbor Alignment (CKNNA): measures similarity between two representations by combining a CKA-style kernel-alignment score with a mutual k-nearest-neighbor restriction, used to track how similar a multimodal model's internal visual-token representations are to a frozen vision foundation model's features across depth.
1 paper
Functional-map spectral alignment
Builds a k-nearest-neighbor graph over shared samples in each of two independently-trained representation spaces, computes the graph Laplacian eigenbasis of each (a spectral analogue of PCA restricted to a similarity graph rather than raw coordinates), then fits a linear operator C between the two truncated spectral bases and decomposes cross-modal compatibility into two independent diagnostics: whether the eigenvalue spectra match (shared manifold complexity) and whether C is diagonal/orthogonal (shared eigenvector orientation).
1 paper
Null-calibrated representational similarity
Corrects any representational-similarity metric (CKA, mutual k-NN, RSA, CCA, Procrustes) for a scale confound by subtracting off its expected value under an explicit null model of independent representations, using permutation-based estimation; global spectral metrics have a non-vanishing O(d/n) null baseline that grows with width/depth, while local-neighborhood metrics have a much smaller O(k/n) baseline.
1 paper
Relative representations (anchor re-coordinatization)
Re-expresses every embedding as its vector of cosine similarities to a fixed set of anchor points from the same space, producing a coordinate system that is provably invariant to rotation, reflection, and rescaling — turning cross-model geometric convergence into a zero-shot stitching capability rather than only a diagnostic similarity number.
1 paper
Triangle-area tri-modal similarity
Replaces pairwise cosine similarity in a contrastive loss with the exact area of the triangle spanned by three unit-norm modality embeddings, computed directly in the shared high-dimensional embedding space via a Gram-determinant (Heron-style) formula -- a single scalar that jointly measures three-way alignment without an anchor modality, a fusion layer, or any pairwise decomposition.
1 paper