CKA recovers layer and network correspondences where CCA metrics fail

measured in 1 paper

Kornblith et al. show CCA-family similarity metrics, invariant to any invertible linear reparameterization, become unreliable when representation dimension is large relative to sample count [kornblith-etal-2019-cka-similarity-revisited] They introduce Centered Kernel Alignment (CKA), invariant only to orthogonal transformation and isotropic scaling [kornblith-etal-2019-cka-similarity-revisited] On sanity checks with Tiny-10 and Plain-(8n+2) CNNs, plus ResNet and Transformer architectures, CKA reliably recovers known layer/network correspondences (e.g. matching layers across different random seeds) that SVCCA and other CCA-based metrics fail to recover [kornblith-etal-2019-cka-similarity-revisited] The comparison is observational, with no causal intervention [kornblith-etal-2019-cka-similarity-revisited]

Structure

Representational-similarity trajectory across depth

Context

Centered Kernel Alignment, orthogonal-invariant similarity, CCA-metric failure mode

Confirmed in models

Convolutional network image classifier (generic) · Convolutional neural network (generic), ResNet (image classifier, various depths) · ResNet, Transformer image classifier (generic) · Transformer (generic)

Method

Centered Kernel Alignment (CKA) representation similarity

Papers

Similarity of Neural Network Representations Revisited — Kornblith, Simon, Norouzi, Mohammad, Lee, Honglak, Hinton, Geoffrey2019 · arXiv:1905.00414