arXiv:1707.08214v2 [cs.CL] 31 Oct 2017

Dual Rectified Linear Units (DReLUs): A Replacement for Tanh Activation
Functions in Quasi-Recurrent Neural Networks
Fréderic Godina, Jonas Degravea, Joni Dambrea, Wesley De Nevea,b
b Center

a IDLab, ELIS, Ghent University – imec, Ghent, Belgium
for Biotech Data Science, Ghent University Global Campus, Incheon, Korea

Abstract
In this paper, we introduce a novel type of Rectified Linear Unit (ReLU), called a Dual Rectified Linear Unit
(DReLU). A DReLU, which comes with an unbounded positive and negative image, can be used as a drop-in
replacement for a tanh activation function in the recurrent step of Quasi-Recurrent Neural Networks (QRNNs)
(Bradbury et al., 2017). Similar to ReLUs, DReLUs are less prone to the vanishing gradient problem, they are noise
robust, and they induce sparse activations.
We independently reproduce the QRNN experiments of Bradbury et al. (2017) and compare our DReLU-based
QRNNs with the original tanh-based QRNNs and Long Short-Term Memory networks (LSTMs) on sentiment classification and word-level language modeling. Additionally, we evaluate on character-level language modeling, showing
that we are able to stack up to eight QRNN layers with DReLUs, thus making it possible to improve the current
state-of-the-art in character-level language modeling over shallow architectures based on LSTMs.

1. Introduction
Rectified activation functions are widely used in
modern neural networks. They are more commonly
known as Rectified Linear Units (ReLUs), which are
essentially neurons with a rectified activation function
(Nair & Hinton, 2010). ReLUs are part of many successful feed-forward neural network architectures such as
ResNets (He et al., 2016) and DenseNets (Huang et al.,
2017), reaching state-of-the-art results in visual recognition tasks. Compared to traditional activation functions
such as sigmoid and tanh, ReLUs offer a number of advantages: (1) they are simple and fast to execute, (2) they
mitigate the vanishing gradient problem, and (3) they induce sparseness. As a result, ReLUs permit training neural networks with up to 1000 layers (He et al., 2016).
However, due to their unboundedness, rectified activation functions have not experienced the same success in Recurrent Neural Networks (RNNs), compared to
the sigmoid and tanh activation functions. The recently
introduced Quasi-Recurrent Neural Network (QRNN)
Preprint submitted to Pattern Recognition Letters

(Bradbury et al., 2017), a hybrid recurrent/convolutional
neural network, can partly solve this issue by avoiding
hidden-to-hidden matrix multiplications. However, using
a rectified activation function instead of a tanh activation
function to calculate a new hidden state is still cumbersome. Indeed, given that ReLUs only have a positive image, strictly negative values cannot be added to a hidden
state, thus making the subtraction of values impossible.
In this paper, we present a new type of neural network
component, called a Dual Rectified Linear Unit (DReLU).
Rather than having a unit with a single activation function,
DReLUs subtract the output of two regular ReLUs, thus
coming with both a positive and negative image. Consequently, similar to ReLUs, DReLUs make it possible
to avoid vanishing gradients and to enforce sparseness.
Moreover, similar to tanh units, DReLUs have a negative
image. However, different from tanh units, DReLUs can
be exactly zero. Indeed, tanh units have difficulties being
exactly zero, often leading to the introduction of noise.
As a result, we can state that DReLUs exhibit properNovember 1, 2017

ties of both ReLUs and tanh units, thus making it possible for DReLUs to replace the functionality of neurons
that make use of a tanh activation function, while at the
same time preventing the occurrence of vanishing gradients when stacking multiple layers. Finally, we also introduce a variant based on the recently introduced Exponential Linear Units (ELUs) (Clevert et al., 2016), called
Dual Exponential Linear Units (DELUs).
We extensively evaluate DELUs and DReLUs on three
different natural language processing tasks, comparing
with the orginal QRNN architecture (Bradbury et al.,
2017): (1) sentiment classification, (2) word-level language modeling, and (3) character-level language modeling.
Our main contributions can be summarized as follows:

ELUpα “ 1q
tanh
ReLU
PReLUpa “ 0.3q

f pxq

1

0

´1
´4

´2

0
x

2

4

Figure 1: Visualization of four important activation functions.

• We introduce a new neural network component, rectified linear activation function. A rectified linear acticalled Dual Rectified Linear Unit (DReLU). This vation function is defined as follows:
newly proposed unit is able to successfully replace units with a tanh activation function in QuasifReL “ maxp0, xq.
(1)
Recurrent Neural Networks (QRNNs).
One particular issue with ReLUs is that the image of the
• We independently reproduce the QRNNs experi- negative domain of the ReLU is always zero. Therements of Bradbury et al. (2017) on sentiment clas- fore, the gradient will not flow through that neuron during
sification and word-level language modeling. Addi- backpropagation. This led to the introduction of Paramettionally, we introduce a third evaluation task, namely ric ReLUs (He et al., 2015), which multiply input values
character-level language modeling. To sustain repro- in the negative domain with a small value, rather than setducibility and transparency, the full implementation ting it to zero. However, the non-saturating behavior in
of all three models is available online (Godin, 2017). the negative domain can introduce noise, while the image
of standard ReLUs is always exactly zero. More recently,
• We demonstrate that we can easily stack up to eight
Exponential Linear Units (ELUs) (Clevert et al., 2016)
DReLU-based QRNN layers without the need of
showed significant improvements over ReLUs. While
skip connections, obtaining state-of-the-art results in
ELUs sacrifice the simplicity of ReLUs, they have a natcharacter-level language modeling.
ural way of combating the bias shift and at the same time
The remainder of the paper is organized as follows. In being noise robust. An exponential linear activation funcSection 2, we give an overview of related work. This tion is defined as follows:
#
overview is followed by a more formal problem definix
if x ą 0
tion and solution in Section 3. In Section 4, we evaluate
(2)
fEL pxq “
α pexppxq ´ 1q
if x ď 0
the proposed solution by means of four different benchmarks. Finally, we end with conclusions in Section 5.
with α ą 0. The parameter α controls the saturation value
of the image for the negative domain.
2. Related Work
Despite the success of ReLUs in Convolutional Neural
Networks (CNNs), ReLUs have not experienced the same
The term Rectified Linear Unit (ReLU) was coined by success in Recurrent Neural Networks (RNNs). Due to
Nair & Hinton (2010). A ReLU is a neuron or unit with a the application of a recurrent step on a possibly infinite
2

sequence, bounded activation functions such as sigmoid
or tanh are needed. Le et al. (2015) managed to train a
simple RNN with a ReLU activation function by carefully
initializing the hidden state weight matrix with the identity matrix. Though good results are obtained, they are
still outperformed by Long Short-Term Memory networks
(LSTMs) (Hochreiter & Schmidhuber, 1997). In general,
ReLUs are used in feed-forward parts of neural networks,
rather than in recurrent parts (Collins et al., 2017).
Due to the variable length of text, RNNs are still the
preferred type of neural network for modeling sequences
of words, and LSTMs in particular (Schmidhuber, 2015).
CNNs are less commonly used because they need large
receptive fields of fixed-input size to capture long-range
dependencies. Only very recently, a number of hybrid approaches have emerged, also leveraging CNNs.
Kalchbrenner et al. (2016) used a fixed-size CNN to perform character-level neural machine translation, obtaining state-of-the-art results. Bradbury et al. (2017) introduced the concept of Quasi-Recurrent Neural Networks
(QRNNs). The two main components of QRNNs are
CNNs and an LSTM-based simplified recurrent step. The
simplified recurrent step allows for modeling sequences
of variable length without using a fixed-size window,
while the combination with CNNs effectuates significant
speed-ups.
All current methods for modeling variable-length sequences still rely on the tanh activation function. This
activation function, however, causes gradients to vanish
during backpropagation. ReLUs are less prone to the vanishing gradient problem but are unbounded and only have
a positive image. Hence, they are less suited for RNNs.
The proposed concept of a DReLU leverages the strengths
of both types of units in order to train models on variablelength sequences.

introduce the concept of DReLUs, as well as the exponential extension, termed DELUs.
3.1. Motivation
RNNs are neural networks that contain cyclic connections. These cyclic connections allow preserving history and are well suited for modeling variable-length sequences. There exist many types of RNNs, most notably
simple RNNs and LSTMs.
In a simple RNN (Elman, 1990), the hidden state at
time step t is defined as follows:
h t “ gpW ¨ ht´1 ` U ¨ xt ` bq,

(3)

in which W and U are weight matrices and b is a bias
vector. The function g is the activation function, which
is in practice a sigmoid or tanh function. The functions
sigmoid and tanh are bounded functions. This implies that
they are functions for which there exists a real number M
such that |gpxq| ď M for all x in domain X. Both the
activation functions sigmoid and tanh are bounded by 1.
Consequently, all hidden states are also bounded and their
absolute value is never larger than 1.
If we simplify the recurrent step defined in Equation 3
to
ht “ gpW ¨ ht´1 q,
(4)
then
ht “ gpW ¨ gpW ¨ ...gpW ¨ h0 qqq.

(5)

If g is an unbounded activation function such as a ReLU,
it becomes clear that the hidden state can grow exponentially large. Indeed, if the largest eigenvalue of the
weight matrix is larger than one, then the norm of the
vector ht will continuously grow and eventually explode
(Goodfellow et al., 2016).
The same holds true for LSTMs for which both the
current cell state and hidden state are dependent on the
previous hidden state via matrix multiplications. Con3. Methodology
sequently, replacing the tanh activation function with an
In this section, we first argue why ReLUs cause train- unbounded activation function would still be problematic,
ing issues in Recurrent Neural Networks (RNNs) such leading to a similar unbounded multiplication as with simas simple RNNs (Elman, 1990) and Long Short-Term ple RNNs.
Memory networks (LSTMs) (Hochreiter & Schmidhuber,
1997). Next, we explain Quasi-Recurrent Neural Net- 3.2. Quasi-recurrent neural networks
works (QRNNs) (Bradbury et al., 2017), which have
QRNNs are hybrid neural networks inspired by
proven to be a viable alternative to LSTMs. Finally, we LSTMs, bringing together the advantages of CNNs and
3

LSTMs (Bradbury et al., 2017). CNNs are fast and highly
parallelizable while LSTMs are able to naturally model
long-range dependencies. However, both simple RNNs
and LSTMs contain a slow recurrent step (cf. Eq. 3),
which can only be executed in sequence. Indeed, ht depends on ht´1 , which depends in its turn on ht´2 , and so
on, thus creating an execution bottleneck.
To reduce the computational effort needed for the recurrent step, the hidden-to-hidden matrix multiplications
of an LSTM were removed and a convolution of size n
over the input x was introduced. Additionally, the input
and forget gates were connected, yielding only two gates,
namely a forget gate ft and an output gate ot . More formally, a QRNN with fo-pooling is defined as follows:
ft , ot “ σpU f,o ¨ xt´n`1:t q,
c̃ t “ tanhpUc ¨ xt´n`1:t q.

(6)
(7)

ct “ ct´1 d ft ` c̃t d p1 ´ ft q.
ht “ ct d ot .

(8)
(9)

To be able to update the cell state with negative values
and, at the same time, use the unbounded rectified linear
activation function, we introduce the novel concept of a
Dual Rectified Linear Unit (DReLU): rather than using a
single ReLU, we propose to subtract two ReLUs.
The dual rectified linear activation function is a twodimensional activation function. In analogy with the rectified linear activation function, the proposed function can
be formally defined as follows:
fDReL pa, bq “ maxp0, aq ´ maxp0, bq

(10)

or
$
0
’
’
’
&a
fDReL pa, bq “
’´b
’
’
%
a´b

bď0
bď0
bą0
bą0
(11)
in which a and b are scalar values. A DReLU can be seen
as being a unit with three separate states. It can be exactly
zero, negative or positive.
An important benefit of DReLUs over tanh activation
functions is the ability to be exactly zero. Indeed, sparseness allows efficient training of larger stacks of neural networks (Glorot et al., 2011) and makes (D)ReLUs more
noise-robust compared to other related activation functions such as Leaky ReLUs.
The partial derivatives of the dual rectified linear activation function with respect to a and b are similar to the
standard rectified linear activation function:
#
0
if a ď 0
B fDReL
(12)
“
Ba
1
if a ą 0

Consequently, the recurrent step is reduced to a simple gated/weighted sum, thus effectuating faster execution. This opens up possibilities for unbounded activation functions such as rectified linear activation functions,
given that the hidden state cannot exponentially grow anymore.
3.3. Dual rectified linear units
As introduced in the previous section, QRNNs do not
contain hidden-to-hidden matrix multiplications anymore,
thus avoiding hidden state explosions when an unbounded
activation function is used. However, replacing the tanh
function of the candidate cell state c̃t with a ReLU (cf.
Eq. 1) would limit the expressiveness of a cell state update
(cf. Eq. 8). Indeed, the values of the gates are bounded
between zero and one. If the activation function of the
candidate cell state c̃t is a ReLU, then the cell state can
only be updated with positive values. Consequently, this
means that only values can be added to the previous cell
state ct´1 , but not subtracted. As a result, the only way
to lower the values of the hidden state is to set the gates
to zero. The tanh function does not have this issue, given
that its image is bounded between minus one and one.
Therefore, the cell state can also be updated with negative
values.

and
B fDReL
“
Bb

#

if
if
if
if

0
´1

aď0
aą0
aď0
aą0

if
if

and
and
and
and

bď0
bą0

(13)

Different from the sigmoid and tanh activation functions,
DReLUs are less prone to vanishing gradients, a property
also shared with ReLUs. When a ReLU or DReLU is active, the magnitude of the gradient through the activation
function is neither amplified nor diminished. Additionally, rectified linear activation functions are fast to execute.
4

Finally, we can replace the calculation of the candidate 4.1.1. Experimental setup
cell state c̃t of a QRNN, which uses a tanh activation funcWe follow the exact same setup as Bradbury et al.
tion (cf. Eq. 7), with a DReLU:
(2017).
All our models are initialized with 300dimensional GloVe word embeddings (Pennington et al.,
c̃t “ maxp0, Uc,1 ¨xt´n`1:t q´maxp0, Uc,2 ¨xt´n`1:t q. (14) 2014) and four stacked recurrent layers. To regularize,
dropout (p “ 0.3) is applied between every layer and L2
3.4. Dual exponential linear units
regularization of 4 ˆ 10´6 is used. As no initialization
ELUs have shown to perform better than ReLUs for im- strategy was specified, we empirically found that using
age classification (Clevert et al., 2016). Indeed, the non- Glorot normal initialization yielded similar results as rezero image of the negative domain combats bias shift and ported by Bradbury et al. (2017).
speeds up learning. However, ELUs contain more comThe IMDb movie review dataset (Maas et al., 2011)
plex calculations and cannot be exactly zero. In analogy contains 25k positive and 25k negative reviews equally diwith DReLUs, we can define DELUs. A dual exponential vided in a set for training and a set for testing. Because no
linear activation function can be expressed as follows:
exact validation set is specified, we split the training set in
five distinct validation sets and run every experiment five
fDEL pa, bq “ fEL paq ´ fEL pbq
(15) times with a different validation set. While Bradbury et al.
(2017) only report experiments with Densely Connected
in which fEL is defined as in Equation 2. fDEL can be per(DC) layers (Huang et al., 2017), we also report results of
ceived as a smoother version of fDReL . Note that although
experiments without DC layers. In this case, densely confEL is only zero when the input is zero, fDEL saturates to
necting all layers means adding concatenative skip conzero when the inputs a and b go to minus infinity.
nections between every layer, except for the final classification layer.
4. Experiments
4.1.2. Discussion
We evaluate the proposed DReLUs and DELUs as part
The mean and standard deviation over five runs for our
of a Quasi-Recurrent Neural Network on three differ- different experiments is reported in Table 1. In general,
ent sequencing tasks. First, we reproduce the results of we can observe only small accuracy differences (0.1%)
(Bradbury et al., 2017) for sentiment classification and between Densely Connected (DC) LSTMs and QRNNs,
word-level language modeling, comparing our DReLUs independent of the activation function used. However,
with tanh-based QRNNs and LSTMs. Second, we evalu- when no skip connections are used, tanh-based QRNNs
ate DReLUs and DELUs on a third task that is more chal- obtain worse accuracy results (90.5) compared to LSTMs
lenging in nature, namely character-level language mod- and DReLU-based QRNNs (90.9 and 91.0, respectively).
eling. The latter consists of two benchmarks.
Moreover, the absolute difference in accuracy between
To support reproducibility, the exact implementation DReLU-based QRNNs and DC-QRNNs is only 0.2%
of all experiments can be found online (Godin, 2017), while for tanh-based QRNNs and DC-QRNNs, this is
including the experiments of the original QRNN paper 0.6%. Indeed, skip connections are typically used to
(Bradbury et al., 2017) for which no full implementation avoid vanishing gradients. ReLU-based activation funcis available.
tions are less prone to vanishing gradients than tanh activation functions, favoring DReLUs over tanh activa4.1. Sentiment Classification
tion functions in QRNNs. Similar conclusions apply for
The first task we consider is sentiment classification of DELUs.
IMDb movie reviews. This is a binary classification task
Our single best model was a DReLU-based QRNN
of documents with up to 2818 tokens. The goal is to com- with an accuracy of 91.6%. However, we believe that
pare our DReLUs and DELUs as part of a QRNN with the a direct comparison with the experimental results of
original tanh-based QRNN model and LSTMs. We also Bradbury et al. (2017) is not meaningful because the valinvestigate the impact of densely connecting all layers.
idation set is unknown.
5

Table 1: Evaluation of QRNNs with DReLU or DELU activation functions, compared to LSTMs and a QRNNs with tanh activation functions for
the task of sentiment classification. The (mean) accuracy and standard deviation is reported. DC means Densely Connected.

Name

DC-LSTM
DC-QRNN
LSTM
QRNN
QRNN

Activation
function

Hidden
state
size

#Params

Bradbury et al. (2017)
tanh
256
3.86M
tanh
256
4.21M

Accuracy

90.9
91.4

Our implementation (Stacked)
tanh
256
2.15M
tanh
300
2.17M

90.9 ˘ 0.2
90.5 ˘ 0.2

DReLU
DELU

91.0 ˘ 0.2
91.0 ˘ 0.1

256
256

2.19M
2.19M

Our implementation (Densely Connected)
tanh
256
91.2 ˘ 0.2
DC-LSTM
3.86M
tanh
242
91.1 ˘ 0.1
DC-QRNN
3.85M
DC-QRNN

DReLU
DELU

200
200

Additionally, we compared our QRNN with an LSTM
and observed a speed-up of 2,5ˆ and 2,1ˆ, for a DReLUand tanh-based QRNN over an LSTM, respectively. Consequently, DReLU-based QRNNs obtain similar results as
LSTMs at more than double the speed, without deploying
skip connections as needed by tanh-based QRNNs.

3.84M
3.84M

91.2 ˘ 0.2
91.1 ˘ 0.2

which uses LSTMs, and (2) Bradbury et al. (2017), which
uses QRNNs. Both papers evaluate their proposed architecture on the Penn Treebank dataset (Marcus et al., 1993)
and use the same setup.
We follow the exact setup of Bradbury et al. (2017) for
training QRNNs with DReLUs and DELUs. That is, a
two-layer stacked QRNN with 640 hidden units equally
sized embeddings and a convolution of size two. For
the QRNN with a tanh activation function, we initialized the weights uniformly in the interval r´0.05; 0.05s.
In QRNNs with rectified activation functions, we initialized the weights randomly, following a normal distribution Np0; 0.1q. For DELUs, we set α to 0.1. For training
the neural network, we use the same training procedure as
described in Bradbury et al. (2017).

4.2. Word-level language modeling

The goal of word-level language modeling is to estimate the conditional probabilities Prpwt |w1 , ..., wt´1 q by
predicting the next word wt given all previous words
rw1 , ..., wt´1 s.
Rather than focusing on obtaining state-of-the-art perplexity results, in this section, we put the emphasis on investigating the relative performance of ReLUs, DReLUs,
and DELUs, compared to stacked LSTMs and stacked
QRNNs with tanh activation functions in similar condi- 4.2.2. Discussion
tions.
The results of our experiments are shown in Table 2.
In general, we can observe that QRNNs with DReLUs
4.2.1. Experimental setup
or DELUs outperform a QRNN with a tanh activation
To properly evaluate our proposed units, we compare function applied in the candidate cell state c̃t . More inwith a reimplementation of (1) Zaremba et al. (2014), terestingly, a QRNN using a single ReLU activation func6

Table 2: Evaluation of different activation functions on the task of word-level language modeling of the PTB corpus.

Name

Activation function

#Parameters

Valid perplexity

Test perplexity

LSTM - medium (Zaremba et al., 2014)
QRNN (Bradbury et al., 2017)

tanh
tanh

20M
18M

86.2
82.9

82.7
79.9

LSTM - medium (our implementation)
QRNN (our implementation)

tanh
tanh

20M
18M

85.8
84.9

82.6
80.0

QRNN (our implementation)

ReLU
DReLU
DELU

18M
20M
20M

89.5
82.6
83.1

85.3
78.4
78.5

Table 3: Activation statistics of the cell states ct in a QRNN for three
different activation functions applied to the candidate cell states c̃t . To
account for near-zero results, we use an interval of s ´ 0.1; 0.1r. The
statistics are calculated on the full test set.

Nearly zero activations
Negative activations
Positive activations

Tanh

ReLU

DReLU

10.02%
45.28%
44.70%

80.22%
0.00%
19.78%

53.90%
23.77%
22.33%

ing able to be exactly zero and induce sparse activations.
Indeed, sparse layer outputs allow for information disentangling and variable-size intermediate representations
(Glorot et al., 2011), and eventually for training large neural networks, in terms of both width and depth.
In this section, we showed that DReLUs and DELUs
have similar properties as tanh activation functions and
that both are suitable replacements. A single ReLU, however, is an inferior replacement for a tanh activation function. Additionally, when using DReLUs, the cell state ct
becomes sparse, which is an important benefit when training large neural networks.

tion performs much worse. The perplexity gap between a
QRNN with a single and a dual ReLU activation function
is 7 perplexity points. Consequently, ReLUs are inferior
replacements for tanh-based units but DReLUs can successfully replace them and even outperform them.
When comparing with a LSTM, following the same
setup as (Zaremba et al., 2014), we can observe that we
outperform a medium-sized LSTM with 650 hidden units.
To analyze the activation pattern the different activation functions induce, we have also calculated the activation statistics of the cell states ct of a QRNN with
a tanh, ReLU and DReLU activation function. The results are depicted in Table 3. While only 10.02% of the
cell states ct is nearly zero when using a tanh activation
function, 80.22% is nearly zero when using a ReLU and
53.9% when using a DReLU. The non-zero cell states ct
for both DReLU and tanh activation functions are equally
divided between positive and negative values. Moreover,
both QRNNs using ReLUs or DReLUs are equally active in the positive domain. Consequently, DReLU activation functions can show similar behavior as tanh activation functions but with the additional benefit of be-

4.3. Character-level language modeling
The final task we consider is character-level language
modeling. The goal is to predict the next character in a
sequence of characters. To that end, we stack up to eight
QRNN layers, either using DReLUs or DELUs, and compare our results with the current state-of-the-art.
4.3.1. Experimental setup
For our character-level language modeling experiments, we consider a small and large more challenging dataset. The small dataset is again the Penn Treebank (PTB) dataset (Marcus et al., 1993) consisting of
roughly 6M characters, while the large dataset is the enwik8/Hutter Prize dataset (Hutter, 2012) which contains
100M characters extracted from Wikipedia. This dataset
is challenging because it contains XML markup and nonLatin characters. We adopt the same train/validation/test
split as Mikolov et al. (2012) for the PTB dataset, and the
90M/5M/5M dataset split for the Hutter Prize dataset.
7

Table 4: Comparison of DReLUs and DELUs with other neural network architectures on the Penn Treebank test set.

Activation
function

Model name
HF-MRNN (Mikolov et al., 2012)
BatchNorm LSTM (Cooijmans et al., 2017)
LayerNorm HM-LSTM (Chung et al., 2017)
LSTM (Ha et al., 2017)
Layer Norm HyperLSTM (Ha et al., 2017)

QRNN (our implementation)

tanh
tanh
tanh
tanh
DELU
DELU
DReLU
DReLU
DReLU
DReLU
DReLU

# Layers

2
2
4
8
4
8
2
4
4
8
8

Hidden state
size

1000
1000
581
579
500
500
250
250
500
250
500

Test BPC

# Params

1.41
1.32
1.24
1.28
1.22
1.26
1.22
1.25
1.21
1.38
1.30
1.25
1.25
1.21

12.26M
14.41M
6.66M
14.69M
6.66M
14.69M
0.82M
1.83M
6.66M
3.85M
14.69M

Table 5: Comparison of DReLUs with other neural network architectures on the Hutter Prize test set.

Model name

Test BPC

Layer Norm LSTM (Ha et al., 2017)
Layer Norm HyperLSTM (Ha et al., 2017)
Layer Norm HM-LSTM (Chung et al., 2017)
ByteNet (Kalchbrenner et al., 2016)
Recurrent Highway Networks (Zilly et al., 2016)

1.40
1.34
1.32
1.31
1.27

QRNN (DReLU - 4 layers - 1000 units)
QRNN (DReLU - 8 layers - 1000 units)

1.32
1.25

Our neural network architecture consists of an embedding layer, a number of QRNN layers, and a final classification layer with a softmax activation function. We
use embeddings of size 50 and hidden states of size 250,
500, and 1000. The number of QRNN layers is 2, 4,
or 8. The width n of the convolution of the first layer
is always six, while at the other layers the convolution
width n is two. The weight matrices are orthogonally
initialized. We trained the neural network using Adam
(Kingma & Ba, 2015) with learning rate 0.0003 and constrained the norm of the gradient at 5. Following Ha et al.
(2017), we used a batch size of 128 and a sequence length
of 100 for the language modeling experiments on the Penn

Treebank dataset. Due to memory constraints, we used
the same batch size of 128 for experiments on the Hutter Prize dataset, but we used a sequence length of 200
instead of 250. We regularized the model using dropout
on the output of every layer (Zaremba et al., 2014) with a
dropout probability of 0.15 for models with hidden state
size 250 and a dropout probability 0.3 for models with
hidden state size 500 and 1000. We applied batch normalization in all QRNN models (Ioffe & Szegedy, 2015;
Cooijmans et al., 2017). The evaluation metric used is
Bits-Per-Character (BPC).

8

in Quasi-Recurrent Neural Networks (QRNNs). Similar
to tanh units, DReLUs have both positive and negative
activations. In addition, DReLUs have several advantages
over tanh units: (1) they do not decrease the magnitude of
gradients when active, (2) they can be exactly zero, making them noise robust, and (3) they induce sparse output
activations in each layer.
We evaluated DReLUs and DELUs on three different tasks. We evaluated on sentiment classification and
showed that DReLUs and DELUs do not need dense
connections to improve gradient backpropagation compared to tanh activation functions when stacking four
layers of QRNNs. We also demonstrated that DReLUs
and DELUs improve perplexity on the task of wordlevel language modeling compared to tanh activation
functions and that a single ReLU is an inferior replacement. Finally, we trained a model with eight stacked
DReLU- and DELU-based QRNN layers and obtained
state-of-the-art results for the task of character-level language modeling on two different datasets, over tanh-based
Hutter Prize experiments. Compared to the PTB dataset, QRNNs and more advanced architectures (Zilly et al.,
the Hutter Prize dataset is a more challenging dataset con- 2016; Kalchbrenner et al., 2016).
sisting of 100M characters, as well as non-Latin characters and XML markup. The results of our experiments
are depicted in Table 5. Apart from the number of lay- Acknowledgments
ers, we trained two identical QRNN-based architectures.
The research activities as described in this paper were
We only consider DReLUs, given that DELUs gave similar results on all other benchmarks. Our model with four funded by Ghent University, imec, Flanders Innovation
layers and 1000 hidden state units in each layer obtains & Entrepreneurship (VLAIO), the Fund for Scientific
a BPC score of 1.32 and performs better than HyperL- Research-Flanders (FWO-Flanders), and the EU.
STMs using a similar number of parameters (25.82M) and
equally good as Hierarchical Multiscale LSTMs (HMLSTM) (Chung et al., 2017). When doubling the num- References
ber of layers to eight, we further reduce the BPC score
Bradbury, J., Merity, S., Xiong, C., & Socher, R. (2017).
to 1.25, outperforming more complicated architectures
Quasi-recurrent neural networks. In International Consuch as ByteNet (Kalchbrenner et al., 2016) and Recurference on Learning Representations (ICLR).
rent Highway Networks (Zilly et al., 2016). Note that the
latter was only evaluated in the context of language mod- Chung, J., Ahn, S., & Bengio, Y. (2017). Hierarchical
eling.
multiscale recurrent neural networks. In Proceedings

4.3.2. Discussion
Penn Treebank experiments. In Table 4, the results of several experiments on the PTB dataset are listed. The first
part shows the results of a number of successful and stateof-the-art character-level language modeling methods, including a vanilla two-layer LSTM, while the second part
shows our results for QRNNs trained with several activation functions.
In general, we can observe that the BPC score of tanhbased QRNNs is worse than the BPC score of DReLU- or
DELU-based QRNNs. DReLUs perform equally good as
DELUs. Doubling the number of QRNN layers from four
to eight is more parameter efficient than doubling the hidden state size from 250 units to 500 units. The language
modeling performance, however, is the same. When using eight QRNN layers and 500 hidden units, we obtain
a BPC score of 1.21, outperforming a two-layer HyperLSTM with a similar number of parameters, a new state-ofthe-art result on this dataset.

of the International Conference on Learning Representations (ICLR).

5. Conclusions

In this paper, we introduced Dual Rectified Linear Clevert, D.-A., Unterthiner, T., & Hochreiter, S. (2016).
Units (DReLUs) and the exponential extension Dual ExFast and accurate deep network learning by exponenponential Units (DELUs), demonstrating that they are
tial linear units (elus). In International Conference on
valid drop-in replacements for the tanh activation function
Learning Representations (ICLR).
9

Collins, J., Sohl-Dickstein, J., & Sussillo, D. (2017). Ca- Hutter, M. (2012). Human knowledge compression conpacity and trainability in recurrent neural networks. In
test. URL: http://prize.hutter1.net/.
International Conference on Learning Representations
Ioffe, S., & Szegedy, C. (2015). Batch normalization:
(ICLR).
Accelerating deep network training by reducing internal covariate shift. In Proceedings of the International
Cooijmans, T., Ballas, N., Laurent, C., & Courville, A. C.
Conference on Machine Learning (ICML) (pp. 448–
(2017). Recurrent batch normalization. In Proceedings
456).
of the 5th International Conference on Learning Representations (ICLR).
Kalchbrenner, N., Espeholt, L., Simonyan, K.,
van den Oord, A., Graves, A., & Kavukcuoglu,
Elman, J. L. (1990). Finding structure in time. Cognitive
K. (2016).
Neural machine translation in
Science, 14, 179–211.
linear time.
CoRR, abs/1610.10099. URL:
Glorot, X., Bordes, A., & Bengio, Y. (2011). Deep sparse
http://arxiv.org/abs/1610.10099.
rectifier neural networks. In Proceedings of the International Conference on Artificial Intelligence and Statis- Kingma, D. P., & Ba, J. (2015). Adam: A method for
stochastic optimization. In Proceedings of the 3rd Intics (AIstats) (pp. 315–323).
ternational Conference on Learning Representations
Godin, F. (2017).
Implementation of quasi(ICLR).
recurrent neural networks using dual rectified
linear units.
In GitHub repository.
URL: Le, Q. V., Jaitly, N., & Hinton, G. E. (2015). A
simple way to initialize recurrent networks of rechttps://github.com/FredericGodin/QuasiRNN-DReLU.
tified linear units. CoRR, abs/1504.00941. URL:
Goodfellow, I., Bengio, Y., & Courville, A.
http://arxiv.org/abs/1504.00941.
(2016).
Deep Learning.
MIT Press.
Maas, A. L., Daly, R. E., Pham, P. T., Huang, D., Ng,
http://www.deeplearningbook.org.
A. Y., & Potts, C. (2011). Learning word vectors for
Ha, D., Dai, A., & Le, Q. (2017). Hypernetworks. In Prosentiment analysis. In Proceedings of the 49th Annual
ceedings of the 5th International Conference on LearnMeeting of the Association for Computational Linguising Representations (ICLR).
tics: Human Language Technologies (ACL-HLT) (pp.
142–150).
He, K., Zhang, X., Ren, S., & Sun, J. (2015). Delving deep into rectifiers: Surpassing human-level per- Marcus, M. P., Marcinkiewicz, M. A., & Santorini, B.
formance on imagenet classification. In IEEE Inter(1993). Building a large annotated corpus of english:
national Conference on Computer Vision (ICCV) (pp.
The penn treebank. Computational Linguistics, 19,
1026–1034).
313–330.
He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep resid- Mikolov, T., Sutskever, I., Deoras, A., Le, H.-S., Komual learning for image recognition. In IEEE Conference
brink, S., & Cernocky, J. (2012). Subword language
on Computer Vision and Pattern Recognition (CVPR)
modeling with neural networks. Preprint (http://www.
(pp. 770–778).
fit. vutbr. cz/imikolov/rnnlm/char. pdf), .
Hochreiter, S., & Schmidhuber, J. (1997). Long short- Nair, V., & Hinton, G. E. (2010). Rectified linear units
term memory. Neural Computation, 9, 1735–1780.
improve restricted boltzmann machines. In Proceedings of the 27th International Conference on Machine
Huang, G., Liu, Z., Weinberger, K. Q., & van der Maaten,
Learning (ICML) (pp. 807–814).
L. (2017). Densely connected convolutional networks.
In Proceedings of the IEEE Conference on Computer Pennington, J., Socher, R., & Manning, C. D. (2014).
Glove: Global vectors for word representation. In
Vision and Pattern Recognition (CVPR).
10

Empirical Methods in Natural Language Processing
(EMNLP) (pp. 1532–1543).
Schmidhuber, J. (2015). Deep learning in neural networks: An overview. Neural Networks, 61, 85 – 117.
Zaremba, W., Sutskever, I., & Vinyals, O.
(2014).
Recurrent neural network regularization.
CoRR, abs/1409.2329. URL:
http://arxiv.org/abs/1409.2329.
Zilly, J. G., Srivastava, R. K., Koutnı́k, J., &
Schmidhuber, J. (2016).
Recurrent highway networks.
CoRR, abs/1607.03474. URL:
http://arxiv.org/abs/1607.03474.

11

