In DeiT-S (and CLIP ViT-B/32) layer-wise sample-wise cosine similarity rises as the compared layers get closer, a pattern the authors explain under an assumed geodesic-curve trajectory

- Jiang, Zhou & Zhu measure layer-wise representation similarity with a sample-wise cosine similarity (COS, computed per sample on the class-token feature of each layer and averaged) and report that it aligns with CKA on DeiT-S, trained from scratch by the authors on CIFAR-10 and on ImageNet-1K (Fig. 2) and, in Fig. 1, an ImageNet-pretrained DeiT-S fine-tuned on CIFAR-10. [jiang-etal-2024-tracing-representation-progression] - Across layers, average COS is almost non-negative, features of far-apart layers are "almost orthogonal", and similarity increases progressively as the two compared layers get closer (a ridge-to-plateau heatmap on the small dataset, a consistent ridge on ImageNet); the same increasing trend is reported for a pretrained CLIP ViT-B/32 within the vision encoder, within the text encoder and across modalities (Appendix D). [jiang-etal-2024-tracing-representation-progression] - Monotonic growth of similarity to the last layer is derived as Theorem 2.1 under an assumed (not tested) geodesic-curve assumption in Wasserstein space, and Theorem 2.2 additionally assumes neural collapse; empirically the layer-wise accuracy of applying the last-layer classifier to hidden layers is highly correlated with COS to the last layer. The geodesic trajectory is therefore a theoretical assumption, not a measured property; the paper's aligned-training method is a separate intervention. [jiang-etal-2024-tracing-representation-progression]