MATH · IN · MODELS

DeiT (Data-efficient image Transformer)

Facebook AI / Meta (Touvron et al. 2021)

Structures found in this family (1)

Representational-similarity trajectory across depth

By model (2)

DeiT-S · 22M
Representational-similarity trajectory across depth
Jiang et al. 2024 DeiT-S trained from scratch (CIFAR-10 / ImageNet-1K) · 22M
Representational-similarity trajectory across depth

Observations (1)

In DeiT-S (and CLIP ViT-B/32) layer-wise sample-wise cosine similarity rises as the compared layers get closer, a pattern the authors explain under an assumed geodesic-curve trajectory

Papers

Tracing Representation Progression: Analyzing and Enhancing Layer-Wise Similarity (2024)