Centered Kernel Alignment (CKA)

methods / Representation Alignment / Centered Kernel Alignment (CKA)

Techniqueintermediate

Continuous statistic comparing two representations' kernel (pairwise-similarity) matrices via a normalized Frobenius inner product between them — the standard quantitative alternative to rank-based mutual-NN alignment for measuring how similar two representations' relative geometry is.

Used in (8 observations)

Twelve trait directions form stable antisocial and prosocial clustersmeasured

structure: Linear Direction · models: Llama-3.1-8B, Qwen2.5-14B, Qwen-2.5-7B, Qwen2.5-0.5B, Llama-3.2-1B · paper: Intrinsic Guardrails: How Semantic Geometry of Personality Interacts with Emergent Misalignment in LLMs

Independent monolingual LMs learn alignable isomorphic geometriesmeasured

structure: Platonic Representation Hypothesis, Linear Subspace · models: Conneau et al. Monolingual MLM (English), Conneau et al. Monolingual MLM (French), Conneau et al. Monolingual MLM (German), Conneau et al. Monolingual MLM (Russian), Conneau et al. Monolingual MLM (Chinese) · paper: Emerging Cross-lingual Structure in Pretrained Language Models

An in-context-RL transformer linearly decodes XY position by layer 2measured

structure: Linear Subspace · models: Custom 3-layer decision-pretraining Transformer (Fang & Rajan 2026) · paper: From Memories to Maps: Mechanisms of In-Context Reinforcement Learning in Transformers

Independently-trained embedding models share linear structure (CKA 0.60-0.88)measured

structure: Linear Subspace · models: text-embedding-3-small, Cohere Embed, Gemini embedding-001, Qwen3-Embedding-8B, E5-mistral-7b-instruct, Llama-3-8B, Qwen2.5-7B-Instruct, Qwen2.5-14B-Instruct, Llama-3.2-1B, OLMo-7B · paper: Characterizing Linear Alignment Across Language Models

GraphCast and Aurora share CKA geometry with a depth-wise scale shiftmeasured

structure: Platonic Representation Hypothesis, Intrinsic-dimension profile across depth · models: GraphCast (weather foundation model), Aurora (weather foundation model) · paper: The physics of AI weather models

After correcting CKA and mutual k-NN alignment for a proven width/depth-dependent null-baseline confound (O(d/n) for global spectral metrics vs O(k/n) for local-neighborhood metrics), the Platonic Representation Hypothesis's global cross-modal convergence largely vanishes while its local-neighborhood convergence survives, across real image-text and video-language pretrained model pairsmeasured

structure: Aristotelian Representation Hypothesis · models: BLOOMZ-560M, BLOOMZ-1.1B, BLOOMZ-1.7B, BLOOMZ-3B, BLOOMZ-7.1B, OpenLLaMA-3B, OpenLLaMA-7B, OpenLLaMA-13B, LLaMA-7B, LLaMA-13B, LLaMA-30B, LLaMA-65B, Gemma-2-9B-it, ViT-Base, ViT-Small, MAE ViT-Base (Masked Autoencoder), DINOv2 ViT-B/14, DINOv2 ViT-L/14, CLIP ViT-B/16, CLIP ViT-L/14, VideoMAE-base, VideoMAE-v2 · paper: Revisiting the Platonic Representation Hypothesis: An Aristotelian View

Two independently-trained JEPA world models converge to a linear isomorphismmeasured

structure: Platonic Representation Hypothesis, Linear Subspace · models: ViT-S/16 (I-JEPA objective, independently trained per view: smallNORB/nuScenes/ImageNet-1k) · paper: Social-JEPA: Emergent Geometric Isomorphism in Independently Trained World Models

Attention builds a shared world-state coordinate system; the MLP specializes itmeasured

structure: Linear Subspace, Attention–MLP Sufficiency Staging Hypothesis · models: Grid-walker decoder transformer (L4/H4/d_model=128, HookedTransformer) · paper: Predictive Statistics Shape Emergent World Representations of Grid Walkers