Toy MLP's pullback metric localizes at class boundaries

measured in 1 paper

- In a toy tanh MLP trained on Boolean functions over a flat-torus embedding, the Riemannian pullback metric of the hidden layer stretches space near class boundaries and compresses it elsewhere, effectively discretizing the input. [brandon-etal-2025-emergent-riemannian-geometry-discrete-computations] - Gaussian curvature peaks and diverges near the class centres (where the embedding circles fold, cos(theta)=0), distinct from where the metric localizes. [brandon-etal-2025-emergent-riemannian-geometry-discrete-computations] - The participation ratio (effective dimensionality) collapses more in the rich regime (SMALL initial weights) than the lazy regime (LARGE initial weights); rich networks also tend to generalize better. [brandon-etal-2025-emergent-riemannian-geometry-discrete-computations] - Analytic/observational on a small toy model (mainly 4 hidden tanh units); init scale is the controlled knob (rich = small-init, lazy = large-init). [brandon-etal-2025-emergent-riemannian-geometry-discrete-computations]

Structure

Decision boundary (as a codimension-1 hypersurface)

Context

Riemannian pullback metric of hidden-layer activations, Gaussian curvature localization near class boundaries, rich vs. lazy training regime, participation ratio as effective-dimensionality statistic

Confirmed in models

Small tanh MLP (AND/OR/XOR on synthetic toroidal/planar input manifolds, rich vs. lazy regimes) · Brandon, Chadwick & Pellegrino (2025) Boolean-Function MLPs

Method

Riemannian pullback-metric curvature analysis, Participation-ratio spectral signal (per-head effective rank over training)

Papers

Emergent Riemannian Geometry over Learning Discrete Computations on Continuous Manifolds — Brandon, Julian, Chadwick, Angus, Pellegrino, Arthur2025 · arXiv:2512.00196