This work presents a new algorithm for training recurrent neural networks (although ideas are applicable to feedforward networks as well). The algorithm is derived from a theory in nonconvex optimization related to the diffusion equation. The contributions made in this work are two fold. First, we show how some seemingly disconnected mechanisms used in deep learning such as smart initialization, annealed learning rate, layerwise pretraining, and noise injection (as done in dropout and SGD) arise naturally and automatically from this framework, without manually crafting them into the algorithms. Second, we present some preliminary results on comparing the proposed method against SGD. It turns out that the new algorithm can achieve similar level of generalization accuracy of SGD in much fewer number of epochs.
@cite_19: When a large feedforward neural network is trained on a small training set, it typically performs poorly on held-out test data. This "overfitting" is greatly reduced by randomly omitting half of the feature detectors on each training case. This prevents complex co-adaptations in which a feature detector is only helpful in the context of several other specific feature detectors. Instead, each neuron learns to detect a feature that is generally helpful for producing the correct answer given the combinatorially large variety of internal contexts in which it must operate. Random "dropout" gives big improvements on many benchmark tasks and sets new records for speech and object recognition.
@cite_13: Denoising auto-encoders (DAEs) have been successfully used to learn new representations for a wide range of machine learning tasks. During training, DAEs make many passes over the training dataset and reconstruct it from partial corruption generated from a pre-specified corrupting distribution. This process learns robust representation, though at the expense of requiring many training epochs, in which the data is explicitly corrupted. In this paper we present the marginalized Denoising Auto-encoder (mDAE), which (approximately) marginalizes out the corruption during training. Effectively, the mDAE takes into account infinitely many corrupted copies of the training data in every epoch, and therefore is able to match or outperform the DAE with much fewer training epochs. We analyze our proposed algorithm and show that it can be understood as a classic auto-encoder with a special form of regularization. In empirical evaluations we show that it attains 1-2 order-of-magnitude speedup in training time over other competing approaches.
As discussed in Section , smoothing can be considered as means to inject noise into the training process. The idea of noise injection is already used in methods such as SGD or dropout @cite_19 in order to improve learning. The key advantage of our framework for noise injection, however, is that the noise injection can be achieved in closed form and without need of sampling. In order words, we can compute the effect of infinitely many noisified objective functions in closed form. This is similar to the idea of Marginalized Denoising Autoencoders (mDA) @cite_13 , where the effect of infinitely many nosified inputs is marginalized to obtain a closed form expression. However, mDA limits the form of the injected noise. Specifically, the marginalized effect is only computable in a c linear reconstruction setup (nonlinearity is applied only after computation of the marginalized reconstruction). In addition, mDA performs noise injection layer by layer in a greedy fashion. In contrast, our framework is able to compute closed form expression for the entire deep network and allowing full nonconvexity of the associated optimization, up to reasonable approximation.