Decision-boundary margins keep shrinking past the error plateau

- Tracking the distance from natural images to the decision boundary across training shows margins keep shrinking well past the point where train and test error have plateaued. [mickisch-etal-2020-decision-boundary-empirical-study] - PGD adversarial training counteracts this, keeping the average L2 margin on MNIST roughly twice as large as standard training. [mickisch-etal-2020-decision-boundary-empirical-study] - Studied on a dense/fully-connected network (MNIST) and a convolutional network (Fashion-MNIST), with CIFAR-10 experiments marked work-in-progress; the paper never uses "LeNet". [mickisch-etal-2020-decision-boundary-empirical-study] - Observational; margins are reported via plots rather than exact tabulated values. [mickisch-etal-2020-decision-boundary-empirical-study]