Latent representations, weights, and neuron gradients become mutually aligned during training, unifying Neural Collapse and the Neural Feature Ansatz

- Ziyin, Chuang, Galanti & Poggio propose the Canonical Representation Hypothesis (CRH), six alignment relations (representation-gradient, representation-weight, gradient-weight; forward and backward) between the covariances of a fully connected layer's latent representations (R), weights (W) and neuron gradients (G), and measure alignment with a Pearson-correlation metric in several from-scratch settings: fully connected nets on synthetic teacher data, ResNet-18 on CIFAR-10, ResNet-18 self-supervised on CIFAR-10/100, and a six-layer eight-head transformer on OpenWebText. [ziyin-etal-2024-formation-of-representations-in-neural-networks] - All six relations are shown to hold strongly in six-layer fully connected nets at large weight decay (at least one forward and one backward relation at small weight decay); in ResNet-18 on CIFAR-10 alignment above 0.7 is reached early in training; in the self-supervised ResNet-18 the representation-gradient alignment holds well for the fully connected layers but less strongly for the convolutional layers; in the 100M-parameter six-layer transformer (measured on the output of each attention head) only the representation-gradient alignment and eigenvalue power-law relations are reported. [ziyin-etal-2024-formation-of-representations-in-neural-networks] - When alignment breaks, eigenvalues of the representation and gradient covariances follow reciprocal power laws (the Polynomial Alignment Hypothesis; exponents observed in [1/3, 3], e.g. exponents 3, 2 and 1 across transformer layers); a minimal-assumption theory (Theorem 2) attributes the regime to the balance between gradient noise and regularization. [ziyin-etal-2024-formation-of-representations-in-neural-networks] - The CRH is proved equivalent to neural collapse for a quasi-interpolating classifier with identity loss-gradient covariance (Theorem 3) and the gradient-weight alignment equals the neural feature ansatz when B is proportional to I; the authors present it as a generalization of neural collapse and an "equivariant NFA", and state it applies only to fully connected layers. [ziyin-etal-2024-formation-of-representations-in-neural-networks]