Published as a conference paper at ICLR 2016

R EGULARIZING RNN S BY S TABILIZING ACTIVATIONS

arXiv:1511.08400v7 [cs.NE] 26 Apr 2016

David Krueger & Roland Memisevic
Department of Computer Science and Operations Research
University of Montreal
Montreal, QC H3T 1J4, Canada
{david.krueger@umontreal.ca, memisevr@iro.umontreal.ca}

A BSTRACT
We stabilize the activations of Recurrent Neural Networks (RNNs) by penalizing
the squared distance between successive hidden states’ norms. This penalty term
is an effective regularizer for RNNs including LSTMs and IRNNs, improving
performance on character-level language modeling and phoneme recognition, and
outperforming weight noise and dropout. We achieve competitive performance
(18.6% PER) on the TIMIT phoneme recognition task for RNNs evaluated without
beam search or an RNN transducer. With this penalty term, IRNN can achieve
similar performance to LSTM on language modeling, although adding the penalty
term to the LSTM results in superior performance. Our penalty term also prevents
the exponential growth of IRNN’s activations outside of their training horizon,
allowing them to generalize to much longer sequences.

1

I NTRODUCTION

Overfitting in machine learning is addressed by restricting the space of hypotheses ( i.e. functions)
considered. This can be accomplished by reducing the number of parameters or using a regularizer
with an inductive bias for simpler models, such as early stopping. More effective regularization
can be achieved by incorporating more sophisticated prior knowledge. Keeping an RNN’s hidden
activations on a reasonable path can be difficult, especially across long time-sequences. With this
in mind, we devise a regularizer for the state representation learned by temporal models, such as
RNNs, that aims to encourage stability of the path taken through representation space. Specifically,
we propose the following additional cost term for Recurrent Neural Networks (RNNs):

β

T
1X
(kht k2 − kht−1 k2 )2
T t=1

Where ht is the vector of hidden activations at time-step t, and β is a hyperparameter controlling the
amounts of regularization. We call this penalty the norm-stabilizer, as it successfully encourages the
norms of the hiddens to be stable (i.e. approximately constant across time). Unlike the “temporal
coherence” penalty of Jonschkowski & Brock (2015), our penalty does not encourage the state
representation to remain constant, only its norm.
In the absence of inputs and nonlinearities, a constant norm would imply orthogonality of the hiddento-hidden transition matrix for simple RNNs (SRNNs). However, in the case of an orthogonal transition matrix, inputs and nonlinearities can still change the norm of the hidden state, resulting in
instability. This makes targeting the hidden activations directly a more attractive option for achieving norm stability. Stability becomes especially important when we seek to generalize to longer
sequences at test time than those seen during training (the “training horizon”).
The hidden state in LSTM (Hochreiter & Schmidhuber, 1997) is usually the product of two squashing nonlinearities, and hence bounded. The norm of the memory cell, however, can grow linearly
when the input, input modulation, and forget gates are all saturated at 1. Nonetheless, we find that
the memory cells exhibit norm stability far past the training horizon, and suggest that this may be
part of what makes LSTM so successful.
1

Published as a conference paper at ICLR 2016

Table 1: LSTM Performance (bits-per-character) on PennTreebank for different values of β.

penalize hidden state
penalize memory cell

β=0
1.47
1.49

β = 50
1.41
1.42

β = 500
1.39
1.40

The activation norms of simple RNNs (SRNNs) with saturating nonlinearities are bounded. With
ReLU nonlinearities, however, activations can explode instead of saturating. When the transition
matrix, Whh has any eigenvalues λ with absolute value greater than 1, the part of the hidden state
that is aligned with the corresponding eigenvector will grow exponentially to the extent that the
ReLU or inputs fails to cancel out this growth.
Simple RNNs with ReLU (Le et al., 2015) or clipped ReLU (Hannun et al., 2014) nonlinearities
have performed competitively on several tasks, suggesting they can learn to be stable. We show,
however, that IRNNs performance can rapidly degrade outside of their training horizon, while the
norm-stabilizer prevents activations from exploding outside of the training horizon allowing IRNNs
to generalize to much longer sequences. Additionally, we show that this penalty results in improved validation performance for IRNNs. Somewhat surprisingly, it also improves performance
for LSTMs, but not tanh-RNNs.
To the best of our knowledge, our proposal is entirely novel. Pascanu et al. (2012) proposed vanishing gradient regularization, which encourages the hidden transition to preserve norm in the direction
of the cost derivative. Like the norm-stabilizer, their cost depends on the path taken through representation space, but the norm stabilzer does not prioritize cost-relevant directions, and accounts
for the effects of inputs as well. A hard constraint (clipping) on the activations of LSTM memory
cells was previously proposed by Sak et al. (2015). Hannun et al. (2014) use a clipped ReLU, which
also has the effect of limiting activations. Both of these techniques operate element-wise however,
whereas we target the activations’ norms. Several other works have used penalties on the difference
of hidden states rather than their norms (Jonschkowski & Brock, 2015; Wen et al., 2015). Other
regularizers for RNNs that do not target norm stability include weight noise (Jim et al., 1996) and
dropout (Pham et al., 2013; Pachitariu & Sahani, 2013; Zaremba et al., 2014).

2

E XPERIMENTS

2.1

C HARACTER -L EVEL L ANGUAGE MODELING ON P ENN T REEBANK

We show that the norm-stabilizer improves performance for character-level language modeling on
PennTreebank (Marcus et al., 1993) for LSTM and IRNNs 1 , but not tanh-RNNs. We present results
for β ∈ {0, 50, 500}. We found that values of β > 500 could slightly improve performance,
but also resulted in much longer training time on this task. Scheduling β to increase throughout
training might allow for faster training. Unless otherwise specified, we use 1000/1600 units for
LSTM/SRNN, and SGD with learning rate=.002, momentum=.99, and gradient clipping=1. We
train for a maximum of 1000 epochs and use sequences of length 50 taken without overlap. When
we encounter a NaN in the cost function, we divide the learning rate by 2, and restart with the
previous epoch’s parameters.
For LSTMs, we either apply the norm-stabilizer penalty only to the memory cells, or only to the
hidden state (in which case we remove the output tanh, as in (Gers & Schmidhuber, 2000)). Although Greff et al. (2015) found the output tanh to be essential for good performance, removing it
gave us a slight improvement in this task. We compare to tanh and ReLU (with and without bias),
with a grid search across cost weight, gradient clipping, and learning rate. For simple RNNs, we
found that the zero-bias ReLU (i.e. TRec (Konda et al., 2014) with threshold 0) gave the best performance. The best performance for ReLU activation functions is obtained with the penalty applied.
For tanh-RNNs, the best performance is obtained without any regularization. Results are better with
the penalty than without for 9 out of 12 experiment settings.
1

As in Le et al. (2015), we initialize Whh to be an identity matrix in our experiments

2

Published as a conference paper at ICLR 2016

Figure 1: Learning Curves for LSTM with different values of β. Penalty is applied to the hidden
state (Left), or the memory cells (Right).

2.1.1

A LTERNATIVE COSTS

We compare 8 alternatives to the norm-stabilizer cost on PennTreeBank for IRNNs without biases
(see Table 3), using the same setup as in 2.1. These include relative error, L1 norm, absolute difference, and penalties that don’t target successive time-steps. The following two penalties performed
2
very poorly and were not included in the table: |∆ kht k2 |, kht k2 .
We find that our proposal of penalizing successive states’ norms gives the best performance, but
some alternatives seem promising and deserve further investigation. In particular, the relative error
could be more appropriate; unlike the norm-stabilizer cost, it cannot be reduced simply by dividing
all of the hidden states by a constant. The value 5 was chosen as a target for the norms based on
the value found by our proposed cost; in practice it would be another hyperparameter to tune. The
success of the other regularizers which encourage (L2 ) norm stability indicates that our inductive
bias in favor of stable norms is useful.
Table 2: Performance with and without norm-stabilizer penalty for different activation functions.
Gradients are clipped at 1 in the first and third, and 106 in the second and fourth columns.

tanh, β = 0
tanh, β = 500
ReLU, β = 0
ReLU, β = 500
TRec, β = 0
TRec, β = 500

lr = .002, gc = 1
1.71
1.57
1.78
1.74
1.62
1.48

lr = .002
1.55
2.70
1.69
1.73
1.63
1.49

lr = .0002, gc = 1
2.15
1.79
1.93
1.65
1.95
1.56

lr = .0002
2.15
1.80
1.93
2.04
1.88
1.56

Table 3: Performance (bits-per-character) of zero-bias IRNN with various penalty terms designed to
encourage norm stability.

β = 50
β = 500

(∆ht )2
1.84
2.19

(∆ kht k2 )2
1.48

(

∆kht k2 2
kht k2 )

(∆ kht k1 )2
2.96
3.18

1.60
1.50

3

(khk2 − 5)2
1.49
1.50

(kh0 k2 − khT k2 )2
3.81
1.54

Published as a conference paper at ICLR 2016

Table 4: Phoneme Error Rate (PER) on TIMIT for different experiment settings, average of 5 experiments. Norm-stabilized networks achieve the best performance. The regularization parameters
are: β - norm stabilizer, p - dropout probability, σ - standard deviation of additive Gaussian weight
noise.

test
dev
2.2

β=0
σ=0
p=0
21.8
19.6

β = 50
σ=0
p=0
20.7
18.6

β = 500
σ=0
p=0
19.0
16.9

β=0
σ = .05
p=0
21.5
19.1

β=0
σ=0
p = .5
21.9
19.5

β = 50
σ=0
p = .5
20.9
18.5

β = 500
σ=0
p = .5
19.4
17.0

β=0
σ = .05
p = .5
21.1
18.9

P HONEME R ECOGNITION ON TIMIT

We show that the norm-stabilizer improves phoneme recognition on the TIMIT dataset, outperforming networks regularized with weight noise and/or dropout. For these experiments, we use a similar
setup to the previous state of the art for an RNN on this task (Graves et al., 2013), with CTC (Graves
et al., 2006) and bidirectional LSTMs with 3 layers of 500 hidden units (for each direction). We
train with Adam (Kingma & Ba, 2014) using learning rate=.001 and gradient clipping=200. Unlike
Graves et al. (2013), we do not use beam search or an RNN transducer. We early stop after 25 epochs
without improvement on the development set.
We apply norm-stabilization to the hidden activations (in this case we do use the output tanh as
is standard) with β ∈ {0, 50, 500}, and use standard deviation .05 for weight noise and p=.5 for
dropout. We try all pair-wise combinations of the regularization techniques. We run 5 experiments
for each of these 10 settings, and report the average phoneme error rate (PER). Combining weight
noise and norm-stabilization gave poor performance, with some networks failing to train, these results are omitted. Adding dropout had a minor effect on results. Norm-stabilized networks had the
best performance (see figure 2 and table 4). Inspired by these results, we decided to train larger
networks with more regularization, and observed further performance improvements (see table 5).
We also used a higher “patience” for our early stopping criterion here, terminating after 100 epochs
without improvement. Unlike previous experiments, we only ran one experiment with each of these
settings. The network with 750 hidden units and β = 1000 gave the best performance on the development set, with dev/test PER of 16.2%/18.6%. This is competitive with the state of the art results
on this task from Graves et al. (2013) and we evaluate without beam search or RNN transducer. although Tóth (2014) achieved 13.9%/16.7% using convolutional neural networks. The network with
1000 hidden units and β = 1000 achieved dev/test PER of 16.7%/17.5%.

Figure 2: Average PER on TIMIT core test set for different combinations or regularizers. The
norm-stabilizer (β) shows a clear positive effect on performance. Weight noise (WN) also improves
performance but less so. Combining weight noise with norm-stabilization gives poor results.
2.3

A DDING TASK

The adding task (Hochreiter & Schmidhuber, 1997) is a toy problem used to test an RNN’s ability
to model long-term dependencies. The goal is to output the sum of two numbers seen at random
4

Published as a conference paper at ICLR 2016

Table 5: Phoneme Error Rate (PER) on TIMIT for experiments with n hidden units and more normstabilizer regularization (β). Networks regularized with weight noise σ = .05 when β = 0. The
network with 750 units and β = 1000 achieved the best dev PER (16.17).

test
dev

β=0
n = 750
21.9
19.6

β = 500
n = 750
18.8
16.8

β = 1000
n = 750
18.6
16.2∗

β = 1500
n = 750
18.0
16.2

β=0
n = 999
21.8
19.1

β = 500
n = 999
19.5
17.4

β = 1000
n = 999
17.5
16.7

β = 1500
n = 999
18.6
16.7

time-steps during training; inputs at other time-steps carry no information. Each element of an input
sequence consists of a pair {n, i}, where n ∈ [0, 1] is chosen at uniform random and i ∈ {0, 1}
indicates which two numbers to add. We use sequences of length 400. In Le et al. (2015), none
of the models were able to reduce the cost below the “short-sighted” baseline set by predicting the
1
first (or second) of the indicated numbers (which gives an expected cost of 12
) for this sequence
length. We are able to solve this task more successfully. We use uniform initialization in [−.01, .01],
learning rate=.01, gradient clipping=1. We compare across nine random seeds with and without the
1
norm-stabilizer (using β = 1). The norm-stabilized networks reduced the test cost below 12
in 8/9
cases, averaging .059 MSE. The unregularized networks averaged .105 MSE, and only outperformed
the short-sighted baseline in 4/9 cases, also failing to improve over a constant predictor in 4/9 cases.
2.4

V ISUALIZING THE EFFECTS OF NORM - STABILIZATION

To test our hypothesis that stability helps networks generalize to longer sequences than they were
trained on, we examined the costs and hidden norms at each time-step.
Comparing identical SRNNs trained with and without norm-stabilizer penalty, we found LSTMs and
RNNs with tanh activation functions continued to perform well far beyond the training horizon. Although the activations of LSTM’s memory cells could potentially grow linearly, in our experiments
they are stable. Applying the norm-stabilizer does significantly decrease their average norm and the
variability of the norm, however (see figure 3). IRNNs, on the other hand, suffered from exploding
activations, resulting in poor performance, but the norm-stabilizer effectively controls the norms
and maintains a high level of performance; see figure 4. Norm-stabilized IRNNs’ performance and
norms were both stable for the longest horizon we evaluated (10,000 time-steps).

Figure 3: Norm (y-axis) of LSTM memory cells (Left) and hidden states (Right) for different values
of β, across time-steps (x-axis). Non-zero values dramatically reduce the mean and variance of the
norms. LSTM memory cells have the potential to grow linearly, but instead exhibit natural stability.

For more insight on why the norm-stabilizer outperforms alternative costs, we examined the hidden
norms of networks trained with values of β ranging from 0 to 200 on a dataset of 1000 length-50
sequences taken from wikipedia (Hutter, 2012). When we penalize the difference of the initial and
final norms, or the difference of the norms from some fixed value, increasing the cost does not change
the shape of the norms; they still begin to explode within the training horizon (see figure 5). For
the norm-stabilizer, however, increasing the penalty significantly delayed (but did not completely
eradicate) activation explosions on this dataset.
5

Published as a conference paper at ICLR 2016

We also noticed that the distribution of activations was more concentrated in fewer hidden units when
applying norm-stabilization on PennTreebank. Similarly, we found that the forget gates in LSTM
networks had a more peaked distribution (see figure 6), while the average across dimensions was
lower (so the network was forgetting more on average at each time step, but a small number of units
were forgetting less). Finally, we found that the eigenvalues of regularized IRNN’s hidden transition
matrices had a larger number of large eigenvalues, while the unregularized IRNN had a much larger
number of eigenvalues closer to 1 in absolute value (see figure 6). This supports our hypothesis that
orthogonal transitions are not inherently desirable in an RNN. By explicitly encouraging stability,
the norm-stabilizer seems to favor solutions that maintain stability via selection of active units, rather
than restricting the choice of transition matrix.

Figure 4: Top: average logarithm of hidden norms as a function of time-step. Bottom: average cost
as a function of time-step. Solid blue - β = 500, dashed red - β = 0. Notice that IRNN’s activations
explode exponentially (linearly in the log-scale) within the training horizon, causing cost quickly go
to infinity outside of the training horizon (50 time-steps).

Figure 5: Hidden norms as a function of time-step for values from 0 to 200 of the norm-stabilizer
(Left and Center) vs. a penalty on the initial and final norms (Right). The norm-stabilizer delays the
explosion of activations by changing the shape of the curve, extending the flat region.

6

Published as a conference paper at ICLR 2016

Figure 6: Left: sorted distribution of average forget-gates for different memory cells in LSTM.
Right: sorted absolute value of eigenvalues of Whh in IRNN. Blue - β = 0, Green - β = 500

3

C ONCLUSION

We introduced norm-based regularization of RNNs to prevent exploding or vanishing activations.
We compare a range of novel methods for encouraging or enforcing norm stability. The best performance is achieved by penalizing the squared difference of subsequent hidden states’ norms. This
penalty, the norm-stabilizer, improved performance on the tasks of language modeling and addition
tasks, and gave state of the art RNN performance on phoneme recognition on the TIMIT dataset.
Future work could involve:
• Exploring the relationship between stability and generative modeling with RNNs
• Applying norm-regularized IRNNs to more challenging tasks
• Applying similar regularization techniques to feedforward nets
ACKNOWLEDGMENTS
This research was developed with funding from the Defense Advanced Research Projects Agency
(DARPA) and the Air Force Research Laborotory (AFRL) . The views, opinions and/or findings
expressed are those of the authors and should not be interpreted as representing the official views or
policies of the Department of Defense or the U.S. Government. We appreciate the many k80 GPUs
provided by ComputeCanada. The authors would like to thank the developers of Theano (Bastien
et al., 2012) and Blocks (van Merriënboer et al., 2015). Special thanks to Alex Lamb, Amar Shah,
Asja Fischer, Caglar Gulcehre, Cesar Laurent, Dmitriy Serdyuk, Dzmitry Bahdanau, Faruk Ahmed,
Harm de Vries, Jose Sotelo, Marcin Moczulski, Martin Arjovsky, Mohammad Pezeshki, Philemon
Brakel, and Saizhen Zhang for useful discussions and/or sharing code.

R EFERENCES
Bastien, Frédéric, Lamblin, Pascal, Pascanu, Razvan, Bergstra, James, Goodfellow, Ian J., Bergeron,
Arnaud, Bouchard, Nicolas, Warde-Farley, David, and Bengio, Yoshua. Theano: new features and
speed improvements. CoRR, abs/1211.5590, 2012. URL http://arxiv.org/abs/1211.
5590.
Gers, Felix A. and Schmidhuber, Jürgen. Recurrent nets that time and count. In IJCNN (3), pp.
189–194, 2000. doi: 10.1109/IJCNN.2000.861302. URL http://dx.doi.org/10.1109/
IJCNN.2000.861302.
Graves, A., Mohamed, A.-R., and Hinton, G. Speech recognition with deep recurrent neural networks. In Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on, pp. 6645–6649, May 2013. doi: 10.1109/ICASSP.2013.6638947.
Graves, Alex, Fernández, Santiago, Gomez, Faustino, and Schmidhuber, Jürgen. Connectionist
temporal classification: Labelling unsegmented sequence data with recurrent neural networks. In
Proceedings of the 23rd International Conference on Machine Learning, ICML ’06, pp. 369–376,
New York, NY, USA, 2006. ACM. ISBN 1-59593-383-2. doi: 10.1145/1143844.1143891. URL
http://doi.acm.org/10.1145/1143844.1143891.
7

Published as a conference paper at ICLR 2016

Greff, Klaus, Srivastava, Rupesh Kumar, Koutnı́k, Jan, Steunebrink, Bas R., and Schmidhuber,
Jürgen. LSTM: A search space odyssey. CoRR, abs/1503.04069, 2015. URL http://arxiv.
org/abs/1503.04069.
Hannun, A., Case, C., Casper, J., Catanzaro, B., Diamos, G., Elsen, E., Prenger, R., Satheesh, S.,
Sengupta, S., Coates, A., and Ng, A. Y. Deep Speech: Scaling up end-to-end speech recognition.
ArXiv e-prints, December 2014.
Hochreiter, Sepp and Schmidhuber, Jürgen. Long short-term memory. Neural Comput., 9(8):1735–
1780, November 1997. ISSN 0899-7667. doi: 10.1162/neco.1997.9.8.1735. URL http://dx.
doi.org/10.1162/neco.1997.9.8.1735.
Hutter, Marcus. The human knowledge compression contest. 2012. URL http://prize.
hutter1.net/.
Jim, Kam-Chuen, Giles, C. Lee, and Horne, Bill G. An analysis of noise in recurrent neural networks: convergence and generalization. IEEE Transactions on Neural Networks, 7(6):1424–1438,
1996. doi: 10.1109/72.548170. URL http://dx.doi.org/10.1109/72.548170.
Jonschkowski, Rico and Brock, Oliver. Learning state representations with robotic priors. Autonomous Robots, 39(3):407–428, 2015. ISSN 0929-5593.
Kingma, Diederik P. and Ba, Jimmy. Adam: A method for stochastic optimization.
abs/1412.6980, 2014. URL http://arxiv.org/abs/1412.6980.

CoRR,

Konda, K., Memisevic, R., and Krueger, D. Zero-bias autoencoders and the benefits of co-adapting
features. ArXiv e-prints, February 2014.
Le, Quoc V., Jaitly, Navdeep, and Hinton, Geoffrey E. A simple way to initialize recurrent networks
of rectified linear units. CoRR, abs/1504.00941, 2015. URL http://arxiv.org/abs/
1504.00941.
Marcus, Mitchell P., Marcinkiewicz, Mary Ann, and Santorini, Beatrice. Building a large annotated
corpus of english: The penn treebank. Comput. Linguist., 19(2):313–330, June 1993. ISSN
0891-2017. URL http://dl.acm.org/citation.cfm?id=972470.972475.
Pachitariu, M. and Sahani, M. Regularization and nonlinearities for neural language models: when
are they needed? ArXiv e-prints, January 2013.
Pascanu, Razvan, Mikolov, Tomas, and Bengio, Yoshua. Understanding the exploding gradient
problem. CoRR, abs/1211.5063, 2012. URL http://arxiv.org/abs/1211.5063.
Pham, Vu, Kermorvant, Christopher, and Louradour, Jérôme. Dropout improves recurrent neural
networks for handwriting recognition. CoRR, abs/1312.4569, 2013. URL http://arxiv.
org/abs/1312.4569.
Sak, Hasim, Senior, Andrew, Rao, Kanishka, Irsoy, Ozan, Graves, Alex, Beaufays, Françoise, and
Schalkwyk, Johan. Learning acoustic frame labeling for speech recognition with recurrent neural networks. In Acoustics, Speech and Signal Processing (ICASSP), 2015 IEEE International
Conference on, pp. 4280–4284. IEEE, 2015.
Tóth, László. Combining time- and frequency-domain convolution in convolutional neural networkbased phone recognition. In IEEE International Conference on Acoustics, Speech and Signal
Processing, ICASSP 2014, Florence, Italy, May 4-9, 2014, pp. 190–194. IEEE, 2014. doi:
10.1109/ICASSP.2014.6853584. URL http://dx.doi.org/10.1109/ICASSP.2014.
6853584.
van Merriënboer, Bart, Bahdanau, Dzmitry, Dumoulin, Vincent, Serdyuk, Dmitriy, Warde-Farley,
David, Chorowski, Jan, and Bengio, Yoshua. Blocks and fuel: Frameworks for deep learning.
CoRR, abs/1506.00619, 2015. URL http://arxiv.org/abs/1506.00619.
Wen, Tsung-Hsien, Gasic, Milica, Mrksic, Nikola, Su, Pei-hao, Vandyke, David, and Young, Steve J.
Semantically conditioned lstm-based natural language generation for spoken dialogue systems.
CoRR, abs/1508.01745, 2015. URL http://arxiv.org/abs/1508.01745.
8

Published as a conference paper at ICLR 2016

Zaremba, Wojciech, Sutskever, Ilya, and Vinyals, Oriol. Recurrent neural network regularization.
CoRR, abs/1409.2329, 2014. URL http://arxiv.org/abs/1409.2329.

9

