Title: Overfitting Mechanism and Avoidance in Deep Neural Networks,
Abstract: Assisted by the availability of data and high performance computing, deep
learning techniques have achieved breakthroughs and surpassed human performance
empirically in difficult tasks, including object recognition, speech
recognition, and natural language processing. As they are being used in
critical applications, understanding underlying mechanisms for their successes
and limitations is imperative. In this paper, we show that overfitting, one of
the fundamental issues in deep neural networks, is due to continuous gradient
updating and scale sensitiveness of cross entropy loss. By separating samples
into correctly and incorrectly classified ones, we show that they behave very
differently, where the loss decreases in the correct ones and increases in the
incorrect ones. Furthermore, by analyzing dynamics during training, we propose
a consensus-based classification algorithm that enables us to avoid overfitting
and significantly improve the classification accuracy especially when the
number of training samples is limited. As each trained neural network depends
on extrinsic factors such as initial values as well as training data, requiring
consensus among multiple models reduces extrinsic factors substantially; for
statistically independent models, the reduction is exponential. Compared to
ensemble algorithms, the proposed algorithm avoids overgeneralization by not
classifying ambiguous inputs. Systematic experimental results demonstrate the
effectiveness of the proposed algorithm. For example, using only 1000 training
samples from MNIST dataset, the proposed algorithm achieves 95% accuracy,
significantly higher than any of the individual models, with 90% of the test
samples classified.