Title: Data Dropout in Arbitrary Basis for Deep Network Regularization,
Abstract: An important problem in training deep networks with high capacity is to
ensure that the trained network works well when presented with new inputs
outside the training dataset. Dropout is an effective regularization technique
to boost the network generalization in which a random subset of the elements of
the given data and the extracted features are set to zero during the training
process. In this paper, a new randomized regularization technique in which we
withhold a random part of the data without necessarily turning off the
neurons/data-elements is proposed. In the proposed method, of which the
conventional dropout is shown to be a special case, random data dropout is
performed in an arbitrary basis, hence the designation Generalized Dropout. We
also present a framework whereby the proposed technique can be applied
efficiently to convolutional neural networks. The presented numerical
experiments demonstrate that the proposed technique yields notable performance
gain. Generalized Dropout provides new insight into the idea of dropout, shows
that we can achieve different performance gains by using different bases
matrices, and opens up a new research question as of how to choose optimal
bases matrices that achieve maximal performance gain.