Class-manifold intrinsic dimension expands then compresses across depth

- In a DeepNet trained on Fashion-MNIST (90.96% test accuracy) and a ResNet trained on CIFAR-10 (87.6%), local and global intrinsic dimension of the per-class representation manifolds first expands in initial layers and then compresses in final layers. [recanatesi-etal-2019-dimensionality-compression-and-expansion] - Representation manifolds are very low-dimensional: on the order of ten dimensions against thousands of neurons per layer. [recanatesi-etal-2019-dimensionality-compression-and-expansion] - ReLU nonlinearities consistently increased dimension by a factor of 1.16 +/- 0.19, while compression came mainly from the weight matrix before the ReLU. [recanatesi-etal-2019-dimensionality-compression-and-expansion] - Compared with the untrained network, training slightly raised local dimension in early layers and strongly lowered it in final layers; doubling DeepNet layer widths changed the dimensions only slightly. [recanatesi-etal-2019-dimensionality-compression-and-expansion] - Noise injected into weights during training deepened the compression in a 7-layer fully connected net; an SGD effective-regularizer argument is given for a two-layer net only. [recanatesi-etal-2019-dimensionality-compression-and-expansion]