Depth-Gated LSTM

arXiv:1508.03790v4 [] 25 Aug 2015

Kaisheng Yao∗
Microsoft Research
kaisheny@microsoft.com
Katerina Vylomova
University of Melbourne

Trevor Cohn
University of Melbourne
tcohn@unimelb.edu.au
Kevin Duh
Nara Institute of Science and Technology

Chris Dyer
Carnegie Mellon University
cdyer@cs.cmu.edu

Abstract
In this short note, we present an extension of long short-term memory (LSTM)
neural networks to using a depth gate to connect memory cells of adjacent layers.
Doing so introduces a linear dependence between lower and upper layer recurrent
units. Importantly, the linear dependence is gated through a gating function, which
we call depth gate. This gate is a function of the lower layer memory cell, the input
to and the past memory cell of this layer. We conducted experiments and verified
that this new architecture of LSTMs was able to improve machine translation and
language modeling performances.

1

Introduction

Deep neural networks (DNNs) have been successfully applied to many areas, including speech [1]
and vision [2]. On natural language processing tasks, recurrent neural networks (RNNs) [3–5] are
widely used because of their ability to memorize long-term dependency.
A typical problem of training deep networks, including RNNs, is gradient diminishing and explosion. This problem is apparent when training a simple RNN. The long short-term memory
(LSTM) [6, 7] neural networks is an extension of simple RNN [3]. In LSTM, a memory cell has
linear dependence of its current activity and its past activity. Importantly, a forget gate is used to
modulate the information flow between the past and the current activities. LSTMs also have input
and output gates to modulate its input and output.
Perhaps the introduction of gating functions in [6, 7] is the most significant improvement to the
recurrent neural networks [3]. More recently, the Gated Recurrent Unit [8] has also adopted the
concept of using gates. LSTMs and GRUs are widely used in many natural language processing
tasks [9, 10].
To construct a deep neural networks, the standard way is to stack many layers of neural networks.
This however has the same problem of building simple recurrent networks. The difference here is
that the error signals from the top, instead of from the last time instance, have to be back-propagated
through many layers of nonlinear transformations and therefore the error signals might be either
diminished or exploded.
∗

Presented at Jelinek Summer Workshop on August 14 2015.

1

Figure 1: LSTM

This short note investigates an extension of LSTMs that uses a depth-gate to connect memory cells
of the lower and upper layers. We review recurrent neural networks in Sec. 2. Section 3 presents
the extension. Experiments are in Sec. 4. We relate this extension with other works in Sec. 5 and
conclude in Sec. 6.

2

Review of recurrent neural networks

A recurrent neural network [3,4] has a hidden state ht that depends on its past value ht-1 recursively;
i.e.,
ht = g(Whh ht-1 + Wxh xt )

(1)

where g(·) is usually a nonlinear function such as tanh. xt is the input. Whh and Wxh are the weight
matrices.
2.1

Long short-term memory (LSTM)

LSTM was initially proposed in [6, 7] and later modified in [11]. We follow the implementation
in [11], which is illustrated in Fig. 1. LSTM introduces a linear dependence between its memory
cells ct and its past ct-1 . Additionally, LSTM has input and output gates. Specially, LSTM is written
below as
it
ft
ct
ot
ht

=
=
=
=
=

σ(Wxi xt + Whi ht-1 + Wci ct-1 )
σ(Wxf xt + Whf ht-1 + Wcf ct-1 )
ft ct-1 + it tanh(Wxc xt + Whc ht-1 )
σ(Wxo xt + Who ht-1 + Wco ct )
ot tanh(ct )

(2)
(3)
(4)
(5)
(6)

where it , ft and ot are input gate, forget gate and output gate of LSTM. ht is the output from the
LSTM. σ(·) is the logistic function. denotes element-wise product. In our application of LSTM,
the forget gate and input gate share the same parameters but are computed as ft = 1 − it . Note that
bias terms are omitted in the above equations but they are applied by default.
2.2

Stacked LSTMs

Typically, LSTMs are stacked to form deep recurrent neural networks, illustrated in the left figure
of Figure 2.
(L)

The output from the lower layer LSTM at layer L is ht . With a possible affine transformation, this
(L+1)
output is used as input xt
in the upper layer LSTM at layer L + 1. Except for this output-input
connection, there is no other connections between the two layers.
2

3

The Depth-gated LSTM

The depth-gated LSTM (DGLSTM) 1 is illustrated in the right figure of Fig. 2. It has a depth gate
(L+1)
that connects the memory cells ct
in the upper layer L + 1 and the memory cell cL
t in the lower
layer L. The depth-gate controls how much flow from the lower memory cell directly to the upper
layer memory cell. The gate function at layer L + 1 at time t is a logistic function as
(L+1)

dt

(L+1)

= σ(bd

(L+1)

(L+1)

+ Wxd

(L+1)

xt

(L+1)

(L+1)

+ wcd

ct-1

(L+1)

+ wld

(L)

ct )

(7)

(L+1)

where bd
is a bias term. Wxd
is the weight matrix to relate depth gate to the input of this
(L+1)
layer. The past memory cell is also related via a weight vector wcd . To relate the lower layer
(L+1)
memory, it uses a weight vector wld . Note that, if lower and upper layer memory cells have
(L+1)
different dimension, wld
should be a matrix instead of a vector.

(a) Stacked LSTM

(b) Depth-gated LSTM

Figure 2: Illustration of the stacked LSTM and the depth-gated LSTM. Notice the additional connection between memory cells in the lower and upper layers in the depth-gated LSTM.
Using the depth gate, a DGLSTM computes the memory cell at layer L + 1 as follows
(L+1)

ct

(L+1)

= dt

(L)

ct

(L+1)

+ ft

(L+1)

ct-1

(L+1)

+ it

(L+1)

(L+1)
tanh(Wxc
xt

(L+1)

+ Whc

(L+1)

ht-1

)(8)

In DGLSTM, equations (2), (3), (5) and (6) are the same as the standard LSTM, except that
DGLSTM uses a superscript L + 1 to denote operations at layer L + 1.
The idea of using gated linear dependence can also be used to connect the first layer memory cell
(1)
(0)
ct with the feature observation xt . In this case, the depth-gate is computed for L = 0 as follows
(1)

dt

(1)

(1)

(1)

= σ(bd + Wxd xt

and the memory cell is computed as


(1)
(1)
(1) (0)
(1)
ct = dt
Wxd xt
+ ft

(1)

(1)

ct-1 + it

(1)

+ wcd

(1)

ct-1 ),

(9)

(0)

(1)
tanh(Wxc
xt

(1)

(1)

+ Whc ht-1 )

1
Implementation
at
https://github.com/kaishengyao/cnn/blob/master/cnn/dglstm.cc
https://github.com/kaishengyao/cnn/blob/master/cnn/dglstm.h.

3

(10)
and

Table 1: BLEU scores in BTEC Chinese to English machine translation task
Depth
3
5
10

GRU
33.95
32.73
30.72

LSTM
32.43
33.52
31.99

DGLSTM
34.48
33.81
32.19

Table 2: BLEU scores by reranking on BTEC Chinese to English machine translation task
Dataset
Dev
Test

4

Baseline
26.61
40.63

DGLSTM
30.05
43.08

Experiments

We applied DGLSTMs on two datasets. The first is BTEC Chinese to English machine translation
task. Its training set consists of 44016 sentence pairs. We use its devset1 and devset 2 for validation,
which in total have 1006 sentence pairs. We use its devset3 for test, which has 506 sentence pairs.
The second dataset is PennTreeBank (PTB) for language modeling. It consists of 42075 sentences
for training, 3371 sentences for development, and 3762 sentences for test.
4.1

Machine translation results

We conducted preliminary experiments and observed that the attention model [9] performed better
than the encoder-decoder method [10]. We therefore applied the attention model [9] in our experiments. Both encoder and decoder used recurrent neural networks in [9]. However, in this experiment, we only used recurrent neural networks for decoder. For encoder, we used word embedding
learned in the training set.
A preliminary experiment showed that the simple RNN [3] performed the worst. We therefore don’t
include the simple RNN results in this paper. We compared DGLSTM with GRU and LSTM. All
these models used 200-dimension hidden layer. We varied the depth of RNNs. Results in Table 1
show that DGLSTM outperforms LSTM and GRU in all of the tested depths.
In another experiment for the machine translation experiment, we used attention model with
DGLSTM to rescore test set k-best lists. We first trained two attention models, one was with 3
layers of DGLSTMs and the other was with 5 layers of DGLSTMs, on training set. Both used
50-dimension hidden layers. We then trained a reranker model using the development data with
100-best lists for each translation pair. The top 100 best lists were generated from the baseline.
The features for the reranker models are the scores from the attention model. The 100-best lists on
the test set were reranked using the trained reranker model. We ran the above described reranking
processes 10 times to get an averaged BLEU scores, which was obtained using one reference. The
BLEU scores are listed in Table 2. Compared to the baseline, DGLSTM improved BLEU scores by
3 points on the Test set.
4.2

Language modeling

We conducted experiments on PTB dataset. We trained a two layer DGLSTM. Each layer has 200
dimension vector. Test set perplexity results are shown in Table 3. Compared against the previously
published results on PTB dataset, DGLSTM obtained the lowest perplexity on PTB test set to our
knowledge.

5

Related works

We developed this method independently in a summer workshop and later knew the works in [13,14].
In highway networks in [13], the output from a layer yt is a linear function to the input xt , in addition
4

Table 3: Penn Treebank Test Set Results.
Model
RNN [5]
LSTM [11]
sRNN [12]
DOT(s)-RNN [12]
DGLSTM

Perplexity
123
117
110
108
96

to the output from a nonlinear path. Both of them are gated as follows
yt = H(xt , Whh ) T(xt , WxT ) + xt C(xt , Wc )
(11)
where T and C are called transform gate and carry gate, respectively. H(·) is the output from
a nonlinear path. Whh , WxT , and Wc are matrices. Therefore, the highway network output has
a direct and linear connection, albeit gated, to the input. This allows highway networks to train
extremely deep networks easily.
DGLSTM is related to the highway networks in using the same idea of linear and gated connection
to input. It differs from the highway networks in not using a specific gate on the non-linear path;
DGLSTM keeps the input and output gates which are applied on the non-linear transformations
in LSTMs. However, the overall effect of using the input and output gates may be similar to the
transfer gate in the highway networks. An additional but important difference is that DGLSTM
linearly connects the memory cells in the lower and upper layers. Because of this, the memory
cell in DGLSTM has errors back-propagated both from the future and from the top layer, linearly
albeit gated. This might be the biggest difference from the highway networks [13] in its current
implementation.
Perhaps the closet work to this research is Grid LSTM [14], which uses LSTMs in different dimensions and connects them using gated linear connections. Because the dimensions can include not
only time, as the typical recurrent neural networks, but also depth and others, Grid LSTM is more
general than DGLSTM, which only considers time and depth. Also, Grid LSTM uses a generic form
of input, memory, and output. Doing so allows a memory cell to have a gated linear dependence not
only on its past memory cell but also on the past observations. Therefore, we consider DGLSTM as
a specific and simple case of Grid LSTM that has gate applied to time and depth only on memory
cells. However, DGLSTM, Grid LSTM and highway networks share the same idea of stacking networks with both linear but gated connections and nonlinear paths. This idea can be applied to fully
connected, convolutional or recurrent layers.

6

Conclusions

We have presented a depth-gated LSTM architecture, which uses a depth-gate to have gated linear
connection between lower and upper layer memory cells. We observed better performances using
this new architecture on machine translation and language modeling tasks. This architecture is
related to the highway networks [13] and Grid LSTM [14] in using an additional linear connection
with gates to regulate information flow across layers.

7

Acknowledgment

We thank Juergen Schmidhuber for pointing out highway networks [13] and grid LSTM [14] works.

References
[1] G. Hinton, L. Deng, D. Yu, G. Dahl, A. r. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke,
P. Nguyen, and T. Sainath and. B. Kingsbury, “Deep neural networks for acoustic modeling in
speech recognition,” 2012.
[2] A Krizhevsky, I Sutskever, and G Hinton, “Imagenet classification with deep convolution
neural networks,” in NIPS, 2012, vol. 25, pp. 1090–1098.
5

[3] J. Elman, “Finding structure in time,” Cognitive science, vol. 14, no. 2, pp. 179–211, 1990.
[4] M. Jordan, “Serial order: a parallel distributed processing approach,” Tech. Rep., Univ. of
California San Diego, 1997.
[5] T. Mikolov, M. Karafiat, L. Burget, J. Cernocky, and S. Khudanpur, “Recurrent neural network
based language model,” in INTERSPEECH, 2010, pp. 1045–1048.
[6] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Computation, vol. 9,
pp. 1735–1780, 1997.
[7] F. A. Gers, J. Schmidhuber, and F. Cummins, “Learning to forget: Continual prediction with
LSTM,” Neural Computation, vol. 12, pp. 2451–2471, 1999.
[8] K. Cho, B. van Merrienboer, D. Bahdanau, and Y. Bengio, “On the properties of neural machine translation,” in arXiv:1409.125, 2014.
[9] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align
and translate,” in arXiv:1409.0473 [cs.CL], 2014.
[10] I. Sutskever, O. Vinyals, and Q.V. Le, “Sequence to sequence learning with neural networks,”
in NIPS, 2014.
[11] A. Graves, “Generating sequences with recurrent neural networks,” in arXiv:1308.0850
[], 2013.
[12] R. Pascanu, C. Gulcehre, K. Cho, and Y. Bengio, “How to construct deep recurrent neural
networks,” in arXiv:1312.6026 [], 2013.
[13] R. K. Srivastava, K. Greff, and Jurgen Schmidhuber,
arxiv:1505.00387v1, May 2015.

“Highway networks,”

in

[14] Nal Kalchbrenner, Ivo Danihelka, and Alex Graves, “Grid long short-term memory,” in
arXiv:1507.01526 [], 2015.

6

