Title: Revisiting Activation Regularization for Language RNNs,
Abstract: Recurrent neural networks (RNNs) serve as a fundamental building block for
many sequence tasks across natural language processing. Recent research has
focused on recurrent dropout techniques or custom RNN cells in order to improve
performance. Both of these can require substantial modifications to the machine
learning model or to the underlying RNN configurations. We revisit traditional
regularization techniques, specifically L2 regularization on RNN activations
and slowness regularization over successive hidden states, to improve the
performance of RNNs on the task of language modeling. Both of these techniques
require minimal modification to existing RNN architectures and result in
performance improvements comparable or superior to more complicated
regularization techniques or custom cell architectures. These regularization
techniques can be used without any modification on optimized LSTM
implementations such as the NVIDIA cuDNN LSTM.