Definition
Given a similarity functional between two representation matrices (rows = examples, columns = neurons/directions) that is invariant to the specific basis each layer happens to use (e.g. canonical correlation analysis (CCA), SVCCA’s SVD-then-CCA composition, or centered kernel alignment (CKA)), this concept describes the trajectory traced out as ranges over a network’s own layers — either compared to a fixed reference layer (e.g. the final layer, or a converged/late-training checkpoint), or pairwise across all layers, or across two independently trained networks’ corresponding layers. Unlike Intrinsic-dimension profile across depth (which tracks a single scalar, the local dimensionality, layer by layer), this concept tracks a relational quantity — how alike two representations are — and the resulting curve/matrix is itself the measured geometric object.
Intuition
Two networks (or two layers of the same network) can each individually look like a normal, reasonably-shaped point cloud, and yet be almost unrelated to each other, or nearly identical up to a change of basis. A similarity trajectory answers “which layers/networks are doing functionally the same thing, geometrically speaking” without requiring either representation to already be aligned to a common coordinate system — which is exactly why CCA-family metrics (invariant to invertible linear transformation) and CKA (invariant to orthogonal transformation and isotropic scaling) are preferred over raw distance comparisons here.
Properties
-
A trajectory, not a single number. The object of interest is the whole curve (or similarity matrix) across depth, not one similarity score — e.g. a curve that stays flat then drops sharply at one layer indicates a genuine architectural transition, which a single averaged similarity score would wash out.
-
Basis-invariant by construction. Because CCA/SVCCA/CKA are invariant to (at least) invertible linear reparameterization of each representation, a low similarity score cannot be explained away as “the two layers just use different coordinate systems for the same information” — genuine representational divergence is what remains once this invariance is accounted for.
-
Distinct from intrinsic-dimension-profile. The dimension profile asks “how many effective coordinates does this one representation need,” entirely intrinsic to a single layer; this concept asks “how related are two representations to each other,” which requires no ID estimation at all and can reveal structure (e.g. two very-differently-shaped-but-highly-correlated layers) invisible to a dimension-only analysis.
-
Foundational precedent: an information-theoretic bottom-up evolution of feature geometry, tracked across depth and training objective. Voita, Sennrich & Titov (2019) trace CCA similarity (against each Transformer’s own final layer) and mutual-information estimates across depth in real trained machine-translation, left-to-right language-model, and masked-language-model Transformers, finding a training-objective-dependent trajectory: LM representations progressively discard “past” token information while building toward the “future” prediction target, while MLM representations first generalize away from surface token identity (peak abstraction in middle layers) before partially reconstructing token identity again near the output. See
voita-etal-2019-cca-and-mutual-information-trajectories-reveal-a-training-objective-dependent-bottom-up-evolution-of-transformer-representation-geometry-across-depth. -
The originating instance: an affine-invariant similarity metric (SVD + CCA) reveals substantial over-parameterization and a bottom-up training-convergence pattern in real trained CNNs and RNNs. Raghu, Gilmer, Yosinski & Sohl-Dickstein (2017, SVCCA) measure effective dimensionality per layer (several layers usable at far lower effective dimension than their nominal width) and show earlier layers stabilize earlier during training than later layers — the founding instance of the CCA-based similarity-trajectory family later extended by Voita et al. (2019, above). See
raghu-etal-2017-svcca-affine-invariant-similarity-reveals-representation-convergence-dynamics-and-effective-dimensionality-in-real-trained-networks. -
A distinct similarity index — invariant to orthogonal transformation and scaling rather than to any invertible linear transform — reliably recovers known correspondences where the CCA family fails. Kornblith, Norouzi, Lee & Hinton (2019, CKA) show CCA-family metrics become unreliable when representation dimension is large relative to sample count, and that Centered Kernel Alignment instead reliably recovers known layer/network correspondences across real trained CNNs of varying width, depth, and initialization. See
kornblith-etal-2019-cka-reliably-recovers-known-layer-and-network-correspondences-in-real-trained-cnns-where-cca-based-metrics-fail. -
A projection-weighted refinement of CCA fixes its sensitivity to low-variance directions, revealing that wider real networks converge to more similar representations and generalizing networks are more mutually similar than memorizing ones. Morcos, Raghu & Bengio (2018, PWCCA) weight each canonical-correlation direction by how much of the original representation it explains, and apply this to real CIFAR-10 convnets and PTB/WikiText-2 LSTM language models across many independent training runs — similarity between independently-trained-but-identical architectures increases with network width, and networks trained on real vs. randomly-shuffled labels (generalizing vs. memorizing solutions) are measurably more self-similar within each group than across groups. See
morcos-etal-2018-pwcca-shows-wider-networks-converge-to-more-similar-real-representations-and-generalizing-networks-are-more-similar-to-each-other-than-to-memorizing-ones. -
VIRAL: MLLM internal visual-token representations diverge from frozen vision-foundation-model features across depth, and an explicit alignment loss causally recovers benchmark performance. Yoon, Jung, Kim et al. (2025) measure CKNNA similarity between real LLaVA-1.5-7B/13B’s internal visual-token representations and frozen VFM (DINOv2/CLIP/SAM/etc.) features across all 32 layers, finding a sharp early-layer drop with only partial mid-layer recovery; adding a training-time cosine-similarity alignment loss causally improves MMVP (28.20%->33.33%), CV-Bench2D (56.82%->59.67%), and POPE (85.70%->88.32%) for LLaVA-1.5-7B+DINOv2. See
yoon-etal-2025-viral-mllm-internal-visual-token-representations-diverge-from-frozen-vision-foundation-model-features-across-depth-and-an-explicit-alignment-loss-causally-recovers-benchmark-performance. -
Trajectory length and curvature, as distinct from a relational similarity curve. Pandey, Singh & Mahdid (2026) measure a related but distinct property of the same layer-by-layer evolution: not how similar two layers’ representations are to each other, but the geometric shape (length, curvature) of the path a population of representations itself traces through ambient space as it moves from layer to layer, across three real trained transformers (GPT-2, TinyLlama, Qwen2.5). Reasoning/analogy prompts produce trajectories of significantly greater curvature than lexical-variation prompts (0.71-0.83 rad vs. 0.27-0.31 rad); semantically related prompts show trajectory convergence peaking at middle-to-late layers (convergence index 0.41-0.58, p<0.001); ambiguous tokens show measurable trajectory bifurcation (up to 5.6x final-layer separation). A layerwise-cosine-similarity component of the same analysis recovers a universal three-phase (encoding/elaboration/output-preparation) structure consistent across all three architectures — this component is the closest to a conventional similarity-trajectory measurement, while trajectory length/curvature is a genuinely distinct, complementary geometric quantity about the path’s own shape rather than a pairwise relational score. Purely observational. See
pandey-etal-2026-real-gpt-2-tinyllama-and-qwen2-5-representation-trajectories-show-higher-curvature-for-reasoning-than-lexical-tasks-and-a-universal-three-phase-layerwise-structure. -
Token-level trajectory curvature and layer-wise entropy across real speech-SSL models, distinguishing distinct optimization regimes. Sadok & Alameda-Pineda (2026, InsideSSL) apply the same trajectory-curvature instrument as Pandey et al. above (curvature of the path traced by adjacent-token transition vectors), but across layers of real pretrained Wav2Vec2, WavLM, HuBERT, and Data2Vec-audio speech models on real LibriSpeech audio: curvature starts high (about 1.4) and decreases to stabilize around 1.2 with depth for most models (manifold “unfolding” toward linear separability), while paired per-layer entropy measurements reveal Wav2Vec2 alone undergoes a sharp entropy collapse at layer 11, coinciding with a spike in an InfoNCE-based invariance-loss measurement — a training-objective-dependent divergence in trajectory shape, paralleling the MT/LM/MLM divergence Voita et al. found in text transformers above, but for the compression (entropy) dimension of the trajectory rather than curvature alone (see also Dimensional collapse). See
sadok-alameda-pineda-2026-real-speech-ssl-models-wav2vec2-wavlm-hubert-and-data2vec-show-distinct-layerwise-entropy-and-trajectory-curvature-regimes-with-wav2vec2-undergoing-a-late-layer-entropy-collapse. -
A “tubularity” trajectory-shape metric (tightness and self-crossings of the layer-to-layer path), applied to a real artificial neural network trained to predict biological visual-cortex responses, distinguishes module-specific trajectory geometry and finds artificial trajectories reorganize more (fewer crossings) than the biological data used only as an external comparison baseline. Bertram, Dyballa, Keller, Kinger & Zucker (2026) build both decoding manifolds (PCA of stimulus-averaged responses) and encoding manifolds (nonnegative tensor factorization plus diffusion maps of neuron x stimulus x time responses) from a real trained foundation model of neural activity (Wang et al.’s FNN, a DenseNet-conv-encoder plus convLSTM plus linear-readout network trained on real mouse visual-cortex MICrONS recordings), finding decoding accuracy rises from 0.59 (early conv encoder) to 0.89 (recurrent module) to 0.88 (readout), while the tubularity metric shows the recurrent module actively “pushes apart” stimulus trajectories (tightness 0.1228, crossings 2.65e-7) with significantly fewer crossings than the biological retina/V1 comparison data (p<0.005, Bonferroni) — a genuinely different trajectory-shape instrument (tightness/self-crossing rather than curvature or CCA/CKA similarity), applied to a scientific/biological-prediction ANN rather than an LLM. See
bertram-etal-2026-a-tubularity-trajectory-shape-metric-in-a-real-ann-trained-to-predict-mouse-visual-cortex-responses-finds-the-recurrent-module-actively-separates-stimulus-trajectories-with-fewer-crossings-than-biological-data.
Exercises
Base
- Why is CCA (or CKA), rather than raw Euclidean distance between two weight-aligned representations, the right tool for comparing two different layers’ representations?
Solution
Two layers can encode identical information while using entirely different, arbitrarily-rotated-and-rescaled coordinate systems for their neurons (there is no reason different layers, let alone different networks, should share a common “meaning” per coordinate axis). Raw Euclidean distance is sensitive to exactly this kind of basis change — it would report two functionally identical representations as maximally dissimilar if their bases differ. CCA and CKA are constructed specifically to be invariant to such reparameterizations (CCA to any invertible linear change of basis, CKA to orthogonal transformations and isotropic scaling), so the reported similarity reflects the actual shared information content rather than an artifact of coordinate choice.
Middle
- A similarity trajectory (against the final layer) is flat and high for a network’s first three layers, then drops sharply at layer four before climbing back to a high value by the final layer. What would this trajectory shape suggest about layer four’s role, and how would you distinguish this from the same phenomenon appearing simply because layer four’s representation has an unusually different intrinsic dimension?
Solution
A sharp mid-trajectory dip suggests layer four performs some transformation that substantially reorganizes the representation relative to both its neighbors and the eventual output — a genuine “bottleneck” or transitional stage, consistent with the MLM finding above (early generalization away from surface form). To rule out that this is simply an artifact of layer four having a very different intrinsic dimension (which could depress a similarity score for reasons unrelated to genuine representational change), one would consult the layer’s own Intrinsic-dimension profile across depth value independently — if the ID profile shows nothing unusual at layer four, the similarity dip is better attributed to a genuine content/organization shift rather than a dimension mismatch.