Last-layer class means converge to a Simplex Equiangular Tight Frame

measured in 1 paper

Papyan, Han & Donoho show that in the terminal phase of training, standard image classifiers (VGG/ResNet/DenseNet) undergo Neural Collapse, with within-class variability collapsing (NC1) [papyan-etal-2020-neural-collapse] Class means become equinorm and maximally equiangular, converging to a Simplex Equiangular Tight Frame (NC2), classifier weights become self-dual with the class means (NC3), and the decision rule converges to nearest-class-center (NC4) [papyan-etal-2020-neural-collapse] All four are demonstrated across three architectures and seven datasets (480 models), and an information-theoretic optimal-code argument proves the Simplex ETF is the unique optimum [papyan-etal-2020-neural-collapse] Yang et al. caveat that NC1's collapse is not complete: a fine-grained information-bearing structure survives inside each cluster, and unsupervised clustering of collapsed CIFAR-10 super-class representations recovers the original 10 classes at 93% [yang-etal-2023-are-neurons-actually-collapsed]

Structure

Polytope (Simplex)

Context

terminal phase of training, within-class variability collapse, Simplex Equiangular Tight Frame, self-dual classifier, nearest-class-center decision rule, information-theoretic optimal code

Confirmed in models

VGG (image classifier, various depths) · VGG, ResNet (image classifier, various depths) · ResNet, DenseNet (image classifier, various depths) · DenseNet

Method

Geometric analysis, Analytical derivation

Papers

Prevalence of Neural Collapse during the terminal phase of deep learning training — Papyan, Vardan, Han, X.Y., Donoho, David L.2020 · arXiv:2008.08186

Are Neurons Actually Collapsed? On the Fine-Grained Structure in Neural Representations — Yang, Yongyi, Steinhardt, Jacob, Hu, Wei2023 · arXiv:2306.17105