Custom Research ResNet (constant-width residual variant, purpose-built for SSL studies)

various (academic)

Structures found in this family (4)

Intrinsic-dimension profile across depth, Linear Separability, Polytope (Simplex), Representation-Weight-Gradient Alignment (Canonical Representation Hypothesis)

By model (5)

Custom RES-L-H ResNet (trained from scratch with VICReg/SimCLR objectives on CIFAR-100/FOOD101) · constant-width residual blocks (Galanti et al. 2022 configuration)

Linear Separability

ResNet classifier (CIFAR-10, 87.6% test accuracy)

Intrinsic-dimension profile across depth

Thrampoulidis et al. 2022 ResNet-18 (step-imbalanced MNIST, Fashion-MNIST and CIFAR10, from scratch)

Polytope (Simplex)

Ziyin et al. 2024 ResNet-18 (CIFAR-10, from scratch)

Representation-Weight-Gradient Alignment (Canonical Representation Hypothesis)

Ziyin et al. 2024 ResNet-18 self-supervised (CIFAR-10/100, from scratch)

Representation-Weight-Gradient Alignment (Canonical Representation Hypothesis)

Observations (4)

Class-manifold intrinsic dimension expands then compresses across depth

SSL training drives emergent semantic-class clustering beyond augmentation invariance

Under class imbalance, class embeddings and classifier weights in the unconstrained-features model interpolate a simplex-encoded label matrix whose SVD factors set their geometry, and convergence to this geometry worsens as imbalance grows

Latent representations, weights, and neuron gradients become mutually aligned during training, unifying Neural Collapse and the Neural Feature Ansatz

Papers

Dimensionality compression and expansion in Deep Neural Networks (2019), Reverse Engineering Self-Supervised Learning (2023), Imbalance Trouble: Revisiting Neural-Collapse Geometry (2022), Formation of Representations in Neural Networks (2024)