Local-complexity density of the ReLU linear-region boundary drops during the terminal phase of training on real MNIST images, coinciding with increased adversarial robustness

Local complexity is defined as the expected local density of the region-boundary ("nonlinear locus") near an input distribution, estimated via a bias-noise perturbation flip-rate estimator, without enumerating the full polyhedral complex [patel-montufar-2024-local-complexity-linear-regions-deep-relu-networks] On a real 4-layer MLP (200 units/layer, 2x-scaled He initialization) trained on a 1,000-image subset of real MNIST digits, both local complexity and total variation drop sharply during the terminal phase of training [patel-montufar-2024-local-complexity-linear-regions-deep-relu-networks] This drop coincides with an increase in adversarial robustness, offered as empirical support for a related finding that the region-boundary nonlinear locus flattens near training data and concentrates near the decision boundary [patel-montufar-2024-local-complexity-linear-regions-deep-relu-networks] Local complexity is also empirically lower for networks trained with larger weight-decay values, though this and the training-dynamics result are purely observational/correlational -- no direct causal intervention on the region arrangement itself is performed [patel-montufar-2024-local-complexity-linear-regions-deep-relu-networks]