MATH · IN · MODELS

Custom Research CNN (purpose-built convolutional net for representation-geometry studies)

various (academic)

Structures found in this family (5)

Circle, Linear region arrangement (polyhedral tessellation of input space), Decision boundary (as a codimension-1 hypersurface), Representational-similarity trajectory across depth, 1D continuum manifold

By model (7)

Custom convolutional net on CIFAR-10 (Carlsson & Brüel Gabrielsson)
Circle
Custom convolutional net on MNIST (Carlsson & Brüel Gabrielsson)
Circle
Custom convolutional net on SVHN (Brüel Gabrielsson & Carlsson)
Circle
CIFAR-10 CNN classifier sub-network, post-relu3 (Gaines & Bi)
Linear region arrangement (polyhedral tessellation of input space)
Convolutional classifier (Fashion-MNIST)
Decision boundary (as a codimension-1 hypersurface)
11-layer CIFAR-10 convnet (Morcos et al. 2018)
Representational-similarity trajectory across depth
8-conv/5-maxpool/1-FC biologically-inspired CNN, trained on ILSVRC2012 ImageNet (Nasr, Viswanathan & Nieder)
1D continuum manifold

Observations (5)

A persistent circle recurs in the convolutional filter weights of real trained CNNs across depths, datasets, and training runs, and fixing an idealized circle as the first layer causally improves cross-dataset generalization
A real ReLU network's connectivity graph of polyhedral linear regions converges toward its 2d average-degree bound, with data-containing regions more connected
Decision-boundary margins keep shrinking past the error plateau
Wider networks converge to more similar representations, measured by PWCCA
Numerosity-tuned units emerge, untrained, in a real CNN trained only for ImageNet object recognition, with Weber-Fechner logarithmic tuning geometry

Papers

Topological Approaches to Deep Learning (2018), Exposition and Interpretation of the Topology of Neural Networks (2019), Characterizing the Discrete Geometry of ReLU Networks (2026), Understanding the Decision Boundary of Deep Neural Networks: An Empirical Study (2020), Insights on Representational Similarity in Neural Networks with Canonical Correlation Analysis (2018), Number detectors spontaneously emerge in a deep neural network designed for visual object recognition (2019)