SAE feature spaces (not raw activations) trained independently on Pythia-70m/160m, Gemma-1/2, and Llama-3/3.1 show high SVCCA and RSA similarity at matched middle layers after activation-correlation feature pairing, far above random-pairing and randomized-weight-model baselines, and semantically matched concept subspaces (e.g. Emotions) are more similar across models than the feature space as a whole

Lan, Torr, Meek, Khakzar, Krueger & Barez (2024) introduce "Analogous Feature Universality": even if independently-trained SAEs learn different individual feature directions, the subspaces spanned by those features may still be similar under rotation-invariant transformations. They first pair SAE features across model pairs by activation correlation, then measure representational similarity of the paired feature spaces via SVCCA (SVD-truncated CCA, mean canonical correlation) and RSA (Pearson-correlation inner similarity, Spearman outer similarity, Euclidean-distance RDMs), applied to SAE decoder weight matrices. Across four main model pairs (Pythia-70m vs Pythia-160m; Gemma-1-2B vs Gemma-2-2B; Gemma-2-2B vs Gemma-2-9B; Llama-3-8B-Instruct vs Llama-3.1-8B), similarity is high at matched middle layers and far exceeds a random-feature-pairing null: e.g. Gemma-2-2B layer 11 vs Gemma-2-9B layer 21 reaches SVCCA=0.70 (random mean 0.009) and RSA=0.195 (random mean 4.38e-4); Llama-3 vs Llama-3.1 (both layer 25) reaches SVCCA=0.3 with 66% mean activation correlation among paired features, retaining 7% of feature pairs after significance filtering; almost all post-layer-0 comparisons yield p-values between 0.00 and 0.01. A second baseline (Appendix D) trains SAEs on a randomized-weight Pythia-70m and finds very low SVCCA/RSA scores with only 1% of feature pairs surviving filtering, confirming the middle-layer similarity is not an artifact of the pairing/scoring procedure itself. Section 4.3 further decomposes similarity by semantic concept category (e.g. "Emotions", "Country"): concept- specific subspaces pass a paired-vs-shuffled-pairing significance test in nearly every category tested (the one reported exception is "Country" for the Gemma-1-2B/Gemma-2-2B pair), meaning some semantically coherent subspaces are more reliably cross-model-similar than the feature space taken as a whole. No causal intervention (feature transplant, steering-vector transfer) is performed; the paper explicitly frames cross-model steering-vector transfer as an unrealized implication of the hypothesis rather than a tested claim. A base-vs-fine-tuned control (Appendix C, Gemma-1-2B vs its Instruct variant, same layer 12) shows even higher similarity (SVCCA=0.84, RSA=0.25) than the cross-architecture pairs, as expected for models sharing a base checkpoint. Son, Rathore, Rufail, Simon, Zhang, Dave, Blondin, Zhu & O'Brien (2025) independently replicate this finding for the identical Gemma-2-2B vs Gemma-2-9B pair with independently-trained SAEs: "we aligned the resulting monosemantic features via activation correlation, and compared the matched feature spaces with metrics such as SVCCA and RSA," reporting paired SVCCA peaking at 0.73 (vs. Lan et al.'s 0.70 for the same pair) and RSA peaking at 0.22 (vs. 0.195), with random-pairing baselines again far lower (0.005-0.034) -- a close, genuinely independent replication of the same model pair and method, not a duplicate submission. They additionally extend the analysis beyond single tokens to multi-token/phrase-level semantic subspaces (e.g. concatenated concept phrases like "happy child"), finding the same convergence pattern holds at the phrase level.