Sampling-based Gradient Regularization for
Capturing Long-Term Dependencies in
Recurrent Neural Networks
Artem Chernodub

?

and Dimitri Nowicki

??

arXiv:1606.07767v3 [cs.NE] 13 Feb 2017

Institute of MMS of NASU, Center for Cybernetics, 42 Glushkova ave., Kiev, Ukraine
03187

Abstract. Vanishing (and exploding) gradients effect is a common problem for recurrent neural networks which use backpropagation method
for calculation of derivatives. We construct an analytical framework to
estimate a contribution of each training example to the norm of the longterm components of the target functions gradient and use it to hold the
norm of the gradients in the suitable range. Using this subroutine we can
construct mini-batches for the stochastic gradient descent (SGD) training that leads to high performance and accuracy of the trained network
even for very complex tasks. To check our framework experimentally we
use some special synthetic benchmarks for testing RNNs on ability to
capture long-term dependencies. Our network can detect links between
events in the (temporal) sequence at the range 100 and longer.

1

Introduction

Recurrent Neural Networks (RNNs) are known as universal approximators of
dynamic systems [1]. Since RNNs are able to simulate any open dynamical system, they have a broad spectrum of applications such as time series forecasting
[2], control of plants [3], language modeling [4], speech recognition, neural machine translation [5] and other domains. The easiest way to create an RNN is
adding the feedback connections to the hidden layer of multilayer perceptron.
This architecture is known as Simple Recurrent Network (SRN). Despite of the
simplicity, it has rich dynamical approximation capabilities mentioned above.
However, in practice training of SRNs using first-order optimization methods
is difficult [6]. The main problem is well-known vanishing/exploding gradients
effect that prevents capturing of long-term dependencies in data. Vanishing gradients effect is a common problem for recurrent and deep neural networks with
sigmoid-like activation functions which uses a backpropagation method for calculation of derivatives. Hochreiter and Schmidhuber designed a set of special
synthetic benchmarks for testing RNNs on ability to capture long-term dependencies [7]. They showed that ordinary SRNs are very ineffective to learn correlations in sequential data if distance between the target events is more than 10
?
??

a.chernodub@gmail.com
nowicki@nnteam.org.ua

2

A.N. Chernodub, D.V. Nowicki

time steps. The solution could be using more advanced second-order optimization
algorithms such as Extended Kalman Filter, LBFGS, Hessian-Free optimization
[8], but they require much more memory and computational resources for stateof-the-art networks. We also mention such an alternative to temporal neural
networks as hierarchical sequence processing with auto-associative memories [9].
The mainstream solution for the gradient control problem is based on more complex architectures such as LSTM [7] or GRU [5] networks. However, training the
SRN’s for catching long-term dependencies is highly desirable at least for better
understanding of underlying processes of the training inside the recurrent and
deep neural networks. Also, SRNs are more compact and fast working models
of RNNs in comparison with LSTMs that is very important for implementation to mobile and embedded devices. Recent research shows the ability to train
SRNs for long term dependencies up to 100 time steps and more using several
new techniques [8], [10]. In this paper we propose a new method to perform the
gradient regularization by selection of proper samples in dataset.

2

Backpropagation Mechanism Revisited

Consider a SRN that at each time step k receives an external input u(k), previous
internal state z(k − 1) and produces output y(k + 1):
a(k) = u(k)win + z(k − 1)wrec + b,
z(k) = f (a(k)),
y(k + 1) = g(z(k)wout ),

(1)

where win is a matrix of input weights, wrec is matrix of recurrent weights,
wout is matrix of output weights, a(k) is known as “presynaptic activations”,
z(k) is a network’s state, f (·) and g(·) are nonlinear activation functions for
hidden and output layer respectively. In this work we always use tanh function
for hidden layer and optionally sof tmax or linear function depending on the
target problem (classification or regression) for output layer.
∂E
=
The dynamic error derivative is a sum of immediate derivatives: ∂w
Ph
∂E
n=1 ∂w(k−n) , where n = 1, ..., h, where h is BPTT’s truncation depth. An
intermediate variable δ ≡ ∂E
∂a called a “local gradients” or simply “deltas” is
usually introduced for convenience,
T
δ(k − h) = δ(k − h + 1)wrec
diag(f 0 (a(k − h))).

Equation (2) may be rewritten using Jacobian matrix J(n) =
δ(k − h) = δ(k − h + 1)J(k − h).

(2)
∂z(n)
∂z(n−1) :

(3)

Now we can use an intuitive understanding of exploding/vanishing gradients
problem that was deeply investigated in classic [6] and modern papers [10]. As it
can be seen from (3), norm of the backpropagated deltas is strongly dependent
on norm of the Jacobians. Moreover, they actually are product of Jacobians:

Sampling-based Gradient Regularization in RNNs

3

δ(k − h) = δ(n)J(k)J(k − 1)...J(k − h + 1). The “older” deltas are, the more
Jacobian matrices were multiplied. If norm of Jacobians are more than 1 if the
gradients will grow exponentially in most cases. It refers to the RNN’s behavior where long-term components are more important than short-term ones. Vice
versa, if norm of Jacobians are less than 1, this leads to vanishing gradients and
“forgetting” the long-term events. In [10] a universal “gradient regularization”
approach that forces the gradient norm to stay in a stable range via modification of the training objective function proposed. However, they used a complex
regularizer to preserve norm in the relevant direction.

3

Differentiation of the gradient’s norm

Let d = {u1 ; t1 ; ...; uN , tN } be a minibatch with ND training examples. We do
forward and back propagation in the network for this minibatch and we get
the difference (correction) vector dw. Letś check how dw influences on gradient
l
vanishing or explosion. Let wrec
be a weight matrix for recurrent layer of the
SRN at the current iteration l of weight update. Suppose we have made the
l
back and forward pass, so wrec
is a correction for the recurrent layer such that
l
l+1
wrec = wrec + dwrec .
(l)
Consider a function S(wrec ) that is equal to squared Euclidian norm of (2)
for iteration l:
1
2
(l)
(4)
S(wrec
) = kδ(k − h, wrec )k2 .
2
(l)

2

Since dwrec is supposed to be small, we can use Taylor expansion of (4)
2
at the current point of weight matrix space:
(l+1)
(l)
(l)
S(wrec
) = S(wrec
+ dS + o( dwrec

2

)).

(5)

2

Lemma 1. Linear term dS in (5) could be expressed as a scalar product of the
auxilary vectors g and dg,
dS = (g, dg),

(6)

where
g=

Q



1
0
i=h diag(f (a(k − i + 1)))wrec
Ph Q1
0
i=1
j=h diag [f (a(k − j +

dg =
v = dwrec , if i = j; v = wrec , if i 6= j.

δ(k),


1))] v δ(k) ,

(7)

Proof.
Using (2), (3) we get δ (l) (k − h):
T
T
δ(k − h) = δ(k)wrec
diag(f 0 (a(k − 1)))...wrec
diag(f 0 (a(k − h + 1))).

(8)

4

A.N. Chernodub, D.V. Nowicki

We introduce a Dn notation as follows:
Dn ≡ diag(f 0 (a(n))).

(9)

Now (8) becomes:
T (l)
T (l)
T (l)
δ (l) (k − h) = δ (l) (k)wrec
Dk−1 wrec
Dk−2 ...wrec
Dk−h .

(10)

Auxillary vector g is a transposed (10):
g = δ (l) (k − h)T .
T

T

(11)

T

Since (AB) = B A , (10) and (11) lead to :
(l)
(l)
(l) (l)
g = Dk−1 wrec
D2 wrec
...Dk−h wrec
δ (k).

Since kxk2 = xT
an equivalent form:

(12)
(l)

2

, regarding (11) and (12), the function S(wrec ) (4) has

1
2
kgk2 .
2
Now we get the differential dS of (13):
(l)
S(wrec
)=

dS = (g, dg),

(13)

(14)

where the vector g in (14) is obtained from (12) that is equivalent (7). Also, we
get vector dg in (14) by differentiating vector g in (12) as follows:
dg =

h
X

(l)
(l)
(l)
δ(k)Dk−h wrec
...Di dwrec
...Dk−1 wrec
,

(15)

i=1

that is the same as (7) up to usage notation Dn . The lemma is proven.
We have to figure out the direction of change of the (Euclidean) norm of
δ(k − h) since the gradient is propagated for h steps back at time step k where
the correction dwrec is used.
Theorem 1. The condition dS > 0 is sufficient to increase the norm of δ (l+1) (k − h)
comparing to δ (l) (k − h) 2 at the next iteration l + 1 of weight correction then
the correction matrix dwrec is used. dS here is defined by (6) and dwrec is con(l+1)
(l)
tained in dS, and wrec = wrec + dwrec . Similarily, dS < 0 is a sufficient
(l+1)
condition for decrease of δ(k − h, wrec ) .
2

Proof.
(l)
(l+1)
Let’s compare values S(wrec ) and S(wrec ) that correspond to current iteration l and next one l + 1 of the weight update. We use Taylor expansion (5)
and the Lemma 1. From (5) follows that the sign of (6) defines a direction of
change for the Euclidean norm of δ(k − h) between the iterations l and l + 1 and
absolute value |dS| defines the magnitude of this change.

2

Sampling-based Gradient Regularization in RNNs

5

Algorithm 1 Algorithm of sampling-based gradient regularization
Input: training data {U, T} , r0 > 0.
for each minibatch ui ; di with ND vectors do
calculate dS (6), if |dS| > 0
continue
make forward and backward propagation
calculate Q(δ, h) (16)
if Q(δ, h) ∈ [Qmin ; Qmax ] then
use current minibatch for training
else
if (Q(δ, h) < Qmin and dS > 0) or (Q(δ, h) > Qmax and dS < 0) then
use current minibatch for training
else
continue
end if
end if
end for

Idea of our sampling-based gradient regularization algorithm is selection of
“proper” samples of data for training. Using Theorem 1 we can clearly find out
an impact of each mini-batch on norm of backpropagated gradients.
We introduce auxiliary variable called Q-factor that measures how much the
norm of the gradient is decreased or increased during the backpropagation. For
ideal catching of long-term dependencies Q-factor must be close to 0.
Q(δ, h) = log10 (

kδ(k)k
).
kδ(k − h)k

(16)

Here we use the simplest and the most straightforward method: we watch a
norm of the gradients; if the norm becomes too small, we omit mini-batches of
data such that decrease this norm. Vice versa, if norm becomes very large, we
skip mini-batches increasing this norm even more. Also, note that it is better
to skip minibatches with large |dS|: they can cause high ”leaps” of the gradient
norm and therefore its self-oscillations.

4

Experiments

We follow [10] and use the following synthetic problems for catching long-term
dependencies: “Adding”, “Multiplication”, “Temporal order”, “Temporal order
3-bit”. Two sets containing 10 SRN with 100 hidden units each were initialized
by random values and saved. Thus, for different training methods initial weights
of neural networks were the same. “Safe” range [Qmin ; Qmax ] for (16) was set to
[−1; 1].
We use SGD optimization, training speed α = 10−5 ...10−3 , momentum µ =
0.9, size of mini-batch is 10. Train / validation / test datasets contains 20,000 /
1000 / 10,000 samples respectively. After each epoch, network’s performance is

6

A.N. Chernodub, D.V. Nowicki

tested on validation dataset; network that has the best performance on the validation dataset is tested on the test dataset, this result is recognized as the final
result. We trained SRNs during 2000 epochs, each epoch consists 50 iterations,
i.e. 100,000 corrections of weights at all.
Weights were initialized by small values from Gaussian distribution with zero
mean and standard deviation σ. On Fig. 1 average norms of gradients as function
of backpropagation depth (before training, further referred as initial gradients)
are graphed for different values σ for the “Temporal order problem”. We see
that good initialization of weights is very important because vanishing/exploding
gradients has monotonous flow in most cases because gradients are propagated
through the same matrix of recurrent weights..

(a) σ= 0.01

(b) σ= 0.005

(c) σ= 0.02

Fig. 1: Average norms of backpropagated (initial) gradients for SRNs, horizon
BPTT h = 100.

Each chart at Fig 1 contains three curves: average norms of local gradients
∂E
δ(k) (blue) and average norms of gradients ∆w(k)in ≡ ∂w
and ∆w(k)rec ≡
in
∂E
∂wrec (red and green). From the graphs at Fig. 3.5 one can ensure on practice
∂E
that to control the norms ∂w
which actually make changes to the weights and
are under the main scope of our interest it is enough to control the norms of
local gradients δ(k) because they are highly correlated.
Finally we used σ = 0.01 as in [10]. However, proper initialization doesn’t
guarantee successful training. Particular case of forward and backward dynamics
(norms of the backpropagated gradients are depicted on the top, mean and
median activation values) during training of SRN network is shown on Fig. 2.
SRN that is depicted on Fig.2 was initialized with σ = 0.01 and initial norms of
backpropagated gradients were similar to Fig. 1 a). However, after 500 iterations
we got norm of gradients less than 10−7 for h = 100. After that almost all the
time neural networks had small gradients in the range 10−7 ...10−8 . From the
graphs on Fig. 2, on the left, we see that area of small gradients is related to
area of saturation for neuron’s activations. This is a symptom of bad network
abilities for successful training and obtaining good generalization properties.

Sampling-based Gradient Regularization in RNNs

(a) Grad. reg. OFF

7

(b) Grad. reg. ON

Fig. 2: Evolution of internal dynamics inside the SRN during training without
gradients regularization (a) and with sampling-based gradients regularization
(b). On both (a) and (b) upper graphs are mean norms in time of backpropagated via BPTT local gradients δ(k); lower ones are mean and median values of
activations a(k).

Using our sampling-based gradients regularization allows to refine the quality
of training (Table 1). For lengths T = 100 and T = 150 improvement is 10-20% in
average. Samples rejected by the algorithm during the training not necessarily
are lost for using in future training process because they may be used when
network is in “safe region” or we may need to change norms of gradients in the
opposite direction.

Acknowledgments
We thank FlyElephant (http://flyelephant.net) and Dmitry Spodarets for computational resources kindly given for our experiments.

5

Conclusion

We provided a novel solution of the problem of exploding and vanishing gradient
effects, applied to the Simple Recurrent Networks. We analytically derived sufficient conditions on increase and decrease the Euclidean vector norm of backpropagated gradients for SRNs. Using this theorem we designed the algorithm that
controls norm of the gradient operating solely with presence of the minbatches in
the training sequence. This framework was tested for long-term prediction on a

8

A.N. Chernodub, D.V. Nowicki

Table 1: Accuracies of trained SRNs for synthetic problems which have longterm dependencies without gradient regularization (traditional training method)
and with sampling-based gradient regularization (proposed method), for T = 100
and 150.

T=100, grad.
reg. OFF
T=100, grad.
reg. ON
T=150, grad.
reg. OFF
T=150, grad.
reg. ON

Adding
Multiplication Temporal order Temporal order 3-bit
best mean best mean best mean
best mean
99%
68% >99% 72%
96%
44%
99%
50%
>99%

96%

>99%

68%

>99%

60%

>99%

62%

34%

11%

N/A

N/A

51%

30%

32%

24%

47%

13%

N/A

N/A

72%

42%

37%

30%

comprehensive set of appropriate benchmarks. Resulting accuracy outperforms
best known SRN learning algorithms by 10-20%. This paradigm could be generalized to deep and multi-layered recurrent networks, that is a subject of our
future research.

References
1. B.G. Horne H.T. Siegelmann. Computational capabilities of recurrent narx neural
networks. IEEE Trans. on Systems, Man, and Cybernetics, B 27(2):208–215, 1997.
2. H. Cardot R. Bone. Advanced Methods for Time Series Prediction Using Recurrent
Neural Networks, chapter Advanced Methods for Time Series Prediction Using
Recurrent Neural Networks, pages 15–36. Intech, Croatia, 2011.
3. D.V. Prokhorov. Toyota prius hev neurocontrol and diagnostics. Neural Networks,
21:458–465, 2008.
4. T Mikolov, M Karafiát, L Burget, J Cernockỳ, and S Khudanpur. Recurrent neural
network based language model. In Interspeech, volume 2, page 3, 2010.
5. D. Bahdanau K. Cho, B. van Merrienboer. On the properties of neural machine
translation: Encoder decoder approaches. In SSST-8, Doha, Qatar, 2014.
6. P. Frasconi Y. Bengio, P. Simard. Learning long-term dependencies with gradient
descent is difficult. IEEE Trans. Neural Networks, 5(2):157–166, 1994.
7. J. Schmidhuber S. Hochreiter, S. Long short-term memory. Neural Computation,
9(8):1735–1780, 1997.
8. I. Sutskever J. Martens. Learning recurrent neural networks with hessian-free
optimization. In Proc. of the ICML, 2011.
9. EM Kussul and DA Rachkovskij. Multilevel assembly neural architecture and
processing of sequences. Neurocomputers and Attention: Connectionism and neurocomputers, 2:577–590, 1991.
10. Y. Bengio R. Pascanu. On the difficulty of training recurrent neural networks.
Technical report, Universite de Montreal, 2012.

