Independently-trained embedding models share linear structure (CKA 0.60-0.88)

measured in 1 paper

Gorbett & Jana measure linear CKA across five independently-trained embedding models (OpenAI text-embedding-3-small, Cohere embed, Gemini embedding-001, Qwen3-Embedding-8B, e5-mistral-7b-instruct), finding similarity 0.595-0.881 [gorbett-jana-2026-characterizing-linear-alignment-across-language-models] In a separate experiment they fit learned affine maps between instruction-tuned generative LMs (Llama-3-8B, Qwen2.5-7B/14B, Llama-3.2-1B, OLMo-7B and others) [gorbett-jana-2026-characterizing-linear-alignment-across-language-models] These maps preserve classification accuracy and OOD-AUROC across model pairs (e.g. 94.5%->93.1%) [gorbett-jana-2026-characterizing-linear-alignment-across-language-models] They enable zero-shot cross-model text generation via a frozen target head, with generation quality correlating with tokenizer exact-match rate (r=0.898) and vocabulary Jaccard overlap (r=0.822) [gorbett-jana-2026-characterizing-linear-alignment-across-language-models]

Structure

Linear Subspace

Context

CKA used to characterize *when* independently trained models are linearly alignable, as a precursor measurement before fitting the alignment map itself, correlating cross-model-generation quality with tokenizer/ vocabulary overlap, isolating a confound that affects transfer quality independent of representational alignment

Confirmed in models

text-embedding-3-small · OpenAI Embeddings, Cohere Embed · Cohere, Gemini embedding-001 · Gemini, Qwen3-Embedding-8B · Qwen, E5-mistral-7b-instruct · E5, Llama-3-8B · Llama, Qwen2.5-7B-Instruct · Qwen, Qwen2.5-14B-Instruct · Qwen, Llama-3.2-1B · Llama, OLMo-7B · OLMo

Method

Centered Kernel Alignment (CKA), Singular Vector Canonical Correlation Analysis (SVCCA)

Papers

Characterizing Linear Alignment Across Language Models — Gorbett, Matthew, Jana, Rajarshi2026 · arXiv:2603.18908