MATH · IN · MODELS

Custom Research CNN (purpose-built convolutional net for representation-geometry studies)

various (academic)

Structures found in this family (7)

Circle, Linear region arrangement (polyhedral tessellation of input space), Decision boundary (as a codimension-1 hypersurface), Representational-similarity trajectory across depth, 1D continuum manifold, Intrinsic-dimension profile across depth, Polytope (Simplex)

By model (9)

Custom convolutional net on CIFAR-10 (Carlsson & Brüel Gabrielsson)
Circle
Custom convolutional net on MNIST (Carlsson & Brüel Gabrielsson)
Circle
Custom convolutional net on SVHN (Brüel Gabrielsson & Carlsson)
Circle
CIFAR-10 CNN classifier sub-network, post-relu3 (Gaines & Bi)
Linear region arrangement (polyhedral tessellation of input space)
Convolutional classifier (Fashion-MNIST)
Decision boundary (as a codimension-1 hypersurface)
11-layer CIFAR-10 convnet (Morcos et al. 2018)
Representational-similarity trajectory across depth
8-conv/5-maxpool/1-FC biologically-inspired CNN, trained on ILSVRC2012 ImageNet (Nasr, Viswanathan & Nieder)
1D continuum manifold
DeepNet convolutional classifier (Fashion-MNIST, 90.96% test accuracy)
Intrinsic-dimension profile across depth
Thrampoulidis et al. 2022 VGG-13 (step-imbalanced MNIST, Fashion-MNIST and CIFAR10, from scratch)
Polytope (Simplex)

Observations (7)

A persistent circle recurs in the convolutional filter weights of real trained CNNs across depths, datasets, and training runs, and fixing an idealized circle as the first layer causally improves cross-dataset generalization
A real ReLU network's connectivity graph of polyhedral linear regions converges toward its 2d average-degree bound, with data-containing regions more connected
Decision-boundary margins keep shrinking past the error plateau
Wider networks converge to more similar representations, measured by PWCCA
Numerosity-tuned units emerge, untrained, in a real CNN trained only for ImageNet object recognition, with Weber-Fechner logarithmic tuning geometry
Class-manifold intrinsic dimension expands then compresses across depth
Under class imbalance, class embeddings and classifier weights in the unconstrained-features model interpolate a simplex-encoded label matrix whose SVD factors set their geometry, and convergence to this geometry worsens as imbalance grows

Papers

Topological Approaches to Deep Learning (2018), Exposition and Interpretation of the Topology of Neural Networks (2019), Characterizing the Discrete Geometry of ReLU Networks (2026), Understanding the Decision Boundary of Deep Neural Networks: An Empirical Study (2020), Insights on Representational Similarity in Neural Networks with Canonical Correlation Analysis (2018), Number detectors spontaneously emerge in a deep neural network designed for visual object recognition (2019), Dimensionality compression and expansion in Deep Neural Networks (2019), Imbalance Trouble: Revisiting Neural-Collapse Geometry (2022)