Title: Manifold Mixup: Learning Better Representations by Interpolating Hidden States,
Abstract: Deep networks often perform well on the data distribution on which they are
trained, yet give incorrect (and often very confident) answers when evaluated
on points from off of the training distribution. This is exemplified by the
adversarial examples phenomenon but can also be seen in terms of model
generalization and domain shift. Ideally, a model would assign lower confidence
to points unlike those from the training distribution. We propose a regularizer
which addresses this issue by training with interpolated hidden states and
encouraging the classifier to be less confident at these points. Because the
hidden states are learned, this has an important effect of encouraging the
hidden states for a class to be concentrated in such a way so that
interpolations within the same class or between two different classes do not
intersect with the real data points from other classes. This has a major
advantage in that it avoids the underfitting which can result from
interpolating in the input space. We prove that the exact condition for this
problem of underfitting to be avoided by Manifold Mixup is that the
dimensionality of the hidden states exceeds the number of classes, which is
often the case in practice. Additionally, this concentration can be seen as
making the features in earlier layers more discriminative. We show that despite
requiring no significant additional computation, Manifold Mixup achieves large
improvements over strong baselines in supervised learning, robustness to
single-step adversarial attacks, semi-supervised learning, and Negative
Log-Likelihood on held out samples.