Delta Networks for Optimized Recurrent Network
Computation
Daniel Neil1 , Jun Haeng Lee1,2 , Tobi Delbruck1 , Shih-Chii Liu1
1

arXiv:1612.05571v1 [cs.NE] 16 Dec 2016

2

Institute of Neuroinformatics, UZH and ETH Zurich, Zurich, Switzerland
Samsung Advanced Institute of Technology, Samsung Electronics, Suwon-Si, Republic of Korea
Email: daniel.l.neil@ini.ethz.ch, junhaeng.lee@gmail.com, tobi@ini.ethz.ch, shih@ini.ethz.ch

Abstract—Many neural networks exhibit stability in their
activation patterns over time in response to inputs from sensors
operating under real-world conditions. By capitalizing on this
property of natural signals, we propose a Recurrent Neural
Network (RNN) architecture called a delta network in which each
neuron transmits its value only when the change in its activation
exceeds a threshold. The execution of RNNs as delta networks
is attractive because their states must be stored and fetched at
every timestep, unlike in convolutional neural networks (CNNs).
We show that a naive run-time delta network implementation
offers modest improvements on the number of memory accesses
and computes, but optimized training techniques confer higher
accuracy at higher speedup. With these optimizations, we demonstrate a 9X reduction in cost with negligible loss of accuracy for
the TIDIGITS audio digit recognition benchmark. Similarly, on
the large Wall Street Journal speech recognition benchmark even
existing networks can be greatly accelerated as delta networks,
and a 5.7x improvement with negligible loss of accuracy can be
obtained through training. Finally, on an end-to-end CNN trained
for steering angle prediction in a driving dataset, the RNN cost
can be reduced by a substantial 100X.
Keywords—Recurrent Neural Networks, Low-precision networks, delta networks.

I.

I NTRODUCTION AND M OTIVATION

Recurrent Neural Networks (RNNs) have achieved tremendous progress in recent years, with the increased availability
of large datasets, more powerful computer resources such as
GPUs, and improvements in their training algorithms. These
combined factors have enabled breakthroughs in the use of
RNNs for processing of temporal sequences. Applications such
as natural language processing [1], speech recognition [2], [3],
and attention-based models for structured prediction [4], [5]
have showcased the advantages of RNNs, as they provide
breakthroughs in former stagnating challenges. RNNs are
attractive because they equip neural networks with memories,
and the introduction of gating units such as long short-term
memory (LSTM) units [6] and gated recurrent units (GRU) [7]
has greatly improved the training process with these networks.
However, RNNs require many matrix-vector multiplications
per layer to calculate the updates of neuron activations over
time.
RNNs also require a large weight memory storage that is
expensive to allocate to on-chip static random access memory.
In a 45nm technology, the energy cost of an off-chip dynamic
32-bit random access memory (SDRAM) access is about 2nJ
and the energy for a 32-bit integer multiply is about 3pJ,
so memory access is about 700 times more expensive than
arithmetic [8]. Architectures can benefit from minimizing this

external memory access. Previous work has focused on a
variety of algorithmic optimizations for reducing compute and
memory access requirements for deep neural networks. These
methods include reduced precision for hardware optimization
([9], [10], [11], [12], [13]); weight encoding, pruning, and
compression ([14], [15]); and architectural optimizations ([16],
[17], [18]). However these studies have not considered temporal properties of the data.
It can be observed that natural inputs to a neural network
tend to have a high degree of temporal autocorrelation, resulting in slowly-changing network states. Fig. 1 demonstrates this
property with a standard convolutional network (VGG-S [19])
operating on a standard video dataset. As seen, the neural representation over time is highly redundant. This slow changing
activation feature is also seen within the computation of RNNs
processing natural inputs, for example, speech (Fig. 2).
Delta networks, as introduced here, exploit the temporal
stability of both the input stream and the associated neural representation to reduce memory access and computation without
loss of accuracy. By caching neuron activations, computations
can be skipped where inputs do not change from the previous
update. Because each neuron that is not updated will save
fetches of entire columns of several weight matrices, efficiently
determining which neurons need to be updated offers significant speedups.
The rest of this paper is organized as follows. Sec. II
introduces the delta network concept in terms of the basic
matrix-vector operations. Sec. III concretely formulates it for a
GRU RNN. Sec. IV proposes a method using a finite threshold
for the deltas that suppresses the accumulation of the transient
approximation error. Sec. V describes methods for optimally
training a delta RNN. Sec. VI shows accuracy versus speedup
for three examples. Finally, Sec. VII compares this work with
other developments and summarizes the results.
II.

D ELTA N ETWORK F ORMULATION

The purpose of a delta network is to transform a dense
matrix-vector multiplication (for example, a weight matrix
and a state vector) into a sparse matrix-vector multiplication followed by a full addition. This transformation leads
to savings on both operations (actual multiplications) and
more importantly memory accesses (weight fetches). Fig. 3
illustrates the savings due to a sparse multiplicative vector.
Zeros are shown with white, while non-zero matrix and vector
values are shown in black. Note the multiplicative effect of
sparsity in the weight matrix and sparsity in the delta vector.

Delta
Vector

Weight Matrix

•

Nonzero Operations

=

Fig. 3. Illustration of saved matrix-vector computation using delta networks
with sparse delta vectors and weight matrices.
Fig. 1. Stability of high-level neural representations over time. The first
1000 frames (40s) from a Hollywood-2 [20] scene recognition clip (the
introduction to American Beauty) are presented to a standard convolutional
network (VGG-S) [19], with the first 50 (arbitrary) features of the top-level
feature vector layer are plotted over time. Note that peaks tend to stay relatively
constant over time, showing network output consistency over time rather than
random feature activation.

Neuron

rt = W (xt − xt−1 ) + rt−1
= W (xt − xt−1 ) + W (xt−1 − xt−2 ) + . . . + r0
= W xt

(3)
(4)
(5)

Thus this formulation, which uses the difference between
two subsequent steps and referred to as the delta network
formulation, can be seen to produce exactly the same result
as the original matrix-vector multiplication.

MFCC

0
5
10
15
20
25
30
35
0
50
100
150
2000

zero as it can be fetched from the previous timestep. Trivially,
x0 = 0 and r0 = 0. It is clear that

A. Theoretical Cost Calculation

50

100

Time

150

200

Fig. 2. Stability in RNN activations over time. The top figure shows the
continually-changing MFCC features for a spoken digit from the TIDIGITS
dataset [21]; the bottom figure shows the corresponding neural network output
activations in response to these features. Note the slow evolution of the
network states over timesteps.

In this example, 20% occupancy of the weight matrix and 20%
occupancy of the ∆ vector requires fetching and computing
only 4% of the original operations.
To illustrate this methodology, consider a general matrixvector multiplication of the form
r = Wx

(1)

that uses n2 compute operations1 , n2 +n reads and n writes for
a W matrix of size n×n and a vector x of size n. Now consider
multiple matrix-vector multiplications for a long input vector
sequence xt indexed by t = 1, 2, . . . , n. The corresponding
result rt can be calculated recursively with:
rt = W ∆ + rt−1 ,

(2)

where ∆ = xt − xt−1 and rt−1 is the result obtained from
the previous calculation; if stored, the compute cost of rt−1 is
1 In this paper, a “compute” operation is either a multiply, an add, or a
multiply-accumulate. The costs of these operations are similar, particularly
when compared to the cost of an off-chip memory operation. See [8] for a
simple comparison of energy costs of compute and memory operations

The ∆ from (2) results in a reduction of computes and
memory accesses of the weight matrix if ∆ is a sparse vector.
To illustrate this, begin by defining oc to be the occupancy
of a vector if a ratio oc of the vector elements are nonzero.
Consider the compute cost for rt ; it consists of the sum of the
cost of calculating ∆ (requiring n operations for a vector of
size n), the cost of adding in the stored previous result rt−1
(n operations), and the cost of the sparse matrix multiply W ∆
(oc · n2 operations for an n × n weight matrix and a sparse
∆ vector of occupancy ratio oc ). Similarly, the memory cost
for calculating rt requires fetching oc · n2 weights for W , 2n
values for ∆, n values for rt−1 and writing out the n values
of the result.
Overall, the compute cost for the standard formulation
(Ccomp,dense ) and the new delta formulation (Ccomp,sparse ) will
be:
Ccomp,dense = n2
Ccomp,sparse = oc · n2 + 2n

(6)
(7)

while the memory access costs for both the standard
(Cmem,dense ) and delta networks (Cmem,sparse ) can be seen
from inspection as:
Cmem,dense = n2 + n
Cmem,sparse = oc · n2 + 4n

(8)
(9)

Thus, the arithmetic intensity (ratio of arithmetic to memory
access costs) as n → ∞ is 1 for both the standard and
delta network methods. This means that for both methods of
calculating rt , every arithmetic operation requires a memory
access, unfortunately placing computational accelerators at a
disadvantage. However, if a sparse occupancy oc of ∆ is

assumed, then the decrease in computes and memory accesses
due to storing the previous state will result in a speedup of:
Cdense /Csparse ≈ n2 /(n2 oc ) = (1/oc )

Mhr,0 = 0, resulting in the following full formulation of the
delta network GRU:
∆x
∆h
Mr,t
Mu,t
Mxc,t
Mhc,t
rt
ut
ct
ht

(10)

For example, if oc = 10%, then the theoretical speedup will
be 10X.
Note that this speedup is determined by the occupancy in
each computed ∆ = xt − xt−1 , implying that this sparsity is
determined by the data stream. Specifically, the regularity with
which values stay exactly the same between xt and xt−1 , or
as demonstrated later, within a certain absolute value called
the threshold, determines the speedup. In a neural network,
the vector x can represents inputs, intermediate activation
values, or outputs of RNNs. If x changes slowly between
subsequent timesteps then the input values xt and xt−1 will
be highly redundant, leading to a low occupancy oc and a
correspondingly increased speedup.
III.

DELTA NETWORK

GRU S

In GRUs, the matrix-vector multiplication operation that
can be replaced with a delta network operation appears several
times, shown in bold below. This GRU formulation is from
[22]:
rt = σr (Wxr xt + Whr ht−1 + br )
ut = σu (Wxu xt + Whu ht−1 + bu )
ct = σc (Wxc xt + rt (Whc ht−1 ) + bc )
ht = (1 − ut ) ht−1 + ut ct

(11)
(12)
(13)
(14)

Here r, u, c and h are reset and update gates, candidate
activation, and activation vectors, respectively, typically a few
hundred elements long. The σ functions are nonlinear logistic
sigmoids that saturate at 0 and 1. The
signifies elementwise multiplication. Each term in bold can be replaced with
the delta update defined in (2), forming:
∆x
∆h
rt
ut
ct
ht

= xt − xt−1
(15)
= ht−1 − ht−2
(16)
= σr (Wxr ∆x + zxr + Whr ∆h + zhr + br )
(17)
= σu (Wxu ∆x + zxu + Whu ∆h + zhu + bu )
(18)
= σc (Wxc ∆x + zxc + rt (Whc ∆h + zhc ) + bc )(19)
= (1 − ut ) ht−1 + ut ct
(20)

where the values zxr , zxu , zxc , zhr , zhu , zhc are recursively
defined as the the stored result of the previous computation
for the input or hidden state, i.e.:
zxr := zxr,t−1 = Wxr (xt−1 − xt−2 ) + zxr,t−2

(21)

The above operation can be applied for the other five values
zxu , zxc , zhr , zhu , zhc . The initial condition at time x0 is z0 :=
0. Also, as can be seen from the equations above, many of
the additive terms, including the stored full-rank pre-activation
states as well as the biases, can be merged into single values
resulting into four stored memory values (Mr , Mu , Mxc , and
Mhr ) for the three gates:
Mt−1 := zx,t−1 + zh,t−1 + b

(22)

Finally, in accordance with the above definitions of the initial
state, the memories M are initialized at their corresponding
biases, i.e., Mr,0 = br , Mu,0 = bu , Mxc,0 = bc , and

IV.

= xt − xt−1
= ht−1 − ht−2
:= Wxr ∆x + Whr ∆h + Mr,t−1
:= Wxu ∆x + Whu ∆h + Mu,t−1
:= Wxc ∆x + Mxc,t−1
:= Whc ∆h + Mhc,t−1
= σr (Mr,t )
= σu (Mu,t )
= σc (Mxc,t + rt Mhc,t )
= (1 − ut ) ht−1 + ut ct

(23)
(24)
(25)
(26)
(27)
(28)
(29)
(30)
(31)
(32)

A PPROXIMATE C ALCULATIONS IN D ELTA N ETWORKS

Note that the formulations described in Secs. II and III are
designed to give precisely the same answer as the original
computation in the network. However, a more aggressive
approach can be taken in the update, inspired by recent studies
that have shown the possibility of greatly reducing weight
precision in neural networks without giving up accuracy [10],
[23]. Instead of skipping a vector-multiplication computation
if a change in the activation ∆ = 0, a vector-multiplication
can be skipped if a value of ∆ is smaller than the threshold
(i.e |∆i,t | < Θ, where Θ is a chosen threshold value for a state
i at time t). That is, if a neuron’s hidden-state M activation
has changed by less than Θ since it was last memorized, the
neuron output will not be propagated, i.e., its ∆ value is set
to zero for that update. Using this threshold, the network will
not produce precisely the same result at each update, but will
produce a result which is approximately correct. Moreover, the
using a threshold substantially increases activation sparsity.
Importantly, if a non-zero threshold is used with a naive
delta change propagation, errors can accumulate over multiple
time steps through state drift. For example, if the input value
xt increases by nearly Θ on every time step, no change will
ever be triggered despite an accumulated significant change
in activation, causing a large drift in error. Therefore, in our
implementation, the memory records the last value causing an
above-threshold change, not the difference since the last time
step.
More formally, we introduce the states x̂i,t−1 and ĥj,t−1 .
These states store the i−th input and the hidden state of the
j−th neurons, respectively, at their last change. The current
input xi,t and state hj,t will be compared against these values
to determine the ∆. Then the x̂i,t−1 and ĥj,t−1 values will
only be updated if the threshold is crossed:

x
if |xi,t − x̂i,t−1 | > Θ
x̂i,t−1 = i,t−1
(33)
x̂i,t−2 otherwise

x − x̂i,t−1 if |xi,t − x̂i,t−1 | > Θ
∆xi,t = i,t
(34)
0
otherwise
(
h
if |hj,t − ĥj,t−1 | > Θ
ĥj,t−1 = j,t−1
(35)
ĥj,t−2 otherwise
(
h − ĥj,t−1 if |hj,t − ĥj,t−1 | > Θ
∆hj,t = j,t
(36)
0
otherwise

That is, when calculating the input delta vector ∆xi,t comprised of each element i at time t, the difference between
two values are used: the current value of the input xi,t ,
and the value the last time the delta vector was nonzero
x̂i,t−1 . Furthermore, if the delta change is under the threshold
Θ, then the delta change is set to zero, producing a small
approximation error that will be corrected when a sufficiently
large change produces a nonzero update. The same formulation
is used for the hidden state delta vector ∆hj,t .
V.

M ETHODS FOR R EDUCING A PPROXIMATION E RROR
AND I NCREASING S PEEDUP

This section presents training methods, constraints, and
optimization schemes that yield faster and more accurate delta
networks.
A. Rounding Network Activations
The thresholded delta network computation described in
Sec. IV performs a rounding function similar to a rounding of
the partially-computed state, since small changes are rounded
to zero while large changes are propagated. Since many
previous investigations have demonstrated methods to train
networks to be robust against small rounding errors by rounding during training, one method that could increase accuracy
is to perform activation rounding. Then, using the techniques
outlined in [23], [10], a network can be successfully trained so
that it is robust to these small rounding errors. Furthermore,
low-precision computation and low-precision parameters can
further reduce power consumption and improve the efficiency
of the network for dedicated hardware implementations.
As explored in previous studies, a low-resolution activation
θL in signed fixed-point format Qm.f with m integer bits
and f fractional bits can be produced from a high-resolution
activation θ by using a deterministic and gradient-preserving
rounding:
θL = round(2f · θ) · 2−f

(37)

with 2f · θ clipped to a range [−2m+f −1 , 2m+f −1 ]. Thus,
the output error cost will incorporate the errors due to small
rounding approximations, and the process of stochastic gradient descent used to increase the accuracy will learn to avoid
these errors through exposure during training.
B. Adding Gaussian Noise to Network Activations
Once thresholding has been introduced, the network must
be robust to the non-propagation of small changes, while
large changes should be considered important. Another way to
provide robustness against small changes is to add Gaussian
noise η to terms that will have a thresholded delta activation:
rt = σr ((xt + ηx )Wxr + (ht−1 + ηh )Whr + br )
(38)
ut = σu ((xt + ηx )Wxu + (ht−1 + ηh )Whu + bu )
(39)
ct = σc ((xt + ηx )Wxc + rt ((ht−1 + ηh )Whc ) + bc )(40)
ht = (1 − ut ) ht−1 + ut ct
(41)
where η ∼ N (µ, σ). That is, η is a vector of samples drawn
from the Gaussian distribution with mean µ and variance σ,
and η ∈ {ηx , ηh }. Each element of these vectors is drawn

independently. Typically, the value µ is set to 0 so that the
expectation is unbiased, e.g., E[xt + ηx ] = E[xt ].
As a result, the Gaussian noise should prevent the network
from being sensitive to minor fluctuations, and increase its
robustness to truncation errors.
C. Training Directly on Delta Networks
However, injecting Gaussian noise at many points in the
network computation is still not the same as the truncation
operation performed by a thresholded delta network. To best
model that truncation, the network should be trained directly
on the errors that arise from a delta network. The resulting
network will then be robust against the types of errors that a
thresholding delta network typically makes.
More accurately, instead of training on the original GRU
equations Eq. 11–14, the state is updated using the delta network model described in Eq. 23–36. This change should incur
no accuracy loss between train accuracy and test accuracy, but
the model may yet have more difficulty during the training if
the model proves harder to optimize and possibly result in an
overall lower accuracy level.
D. Considering Additional Speedup from Weight Sparsity
Furthermore, the speedup from using a delta network so far
has been considered to only arise from the sparse delta vectors
that allow skipping columns of the weight matrices. However,
the amount of sparsity in the weight matrices of deep networks
after training also can affect the savings in the computational
cost and the speedup. Studies such as in [24] show that
in trained low-precision networks, the weight matrices can
be quite sparse. For example, in a ternary or 3-bit weight
network the weight matrix sparsity can exceed 80% for small
RNNs. Since every nonzero input vector element is multiplied
by a column of the weight matrix, this computation can be
skipped if the weight value is zero. That is, the zeros in
the weight matrix act multiplicatively with the delta vector
to produce even fewer necessary multiply-accumulates, as
illustrated above in Fig. 3. The calculation of the matrix-vector
product then costs:
Ccomp,sparse = om · oc · n2 + 2n
Cmem,sparse = om · oc · n2 + 4n

(42)
(43)

for a weight matrix with occupancy om . By comparison to
Eq. 10, the system can achieve a theoretical speedup of
1/(om · oc ). That is, by compressing the weight matrix and
only fetching nonzero weight elements that combine with the
nonzero state vector, a higher speedup can be obtained without
degrading the accuracy.
E. Incurring Sparsity Cost on Changes in Activation
Finally, if the network is trained using the delta network
model, a cost can be associated with the delta terms and added
into the overall cost. In a batch of input samples, the L1
norm for ∆h can be calculated as the mean absolute delta
changes, and this norm can be scaled by a weighting factor β.
This Lsparse cost can then be additively incorporated into the
standard loss function. That is:
Lsparse = β||∆h||1

(44)

99
98

1ab
2ab

1a

100

14

2, 2a

12

98

97

8

94

6

93

Accuracy [%]

1

95

96

1

94

4

92

1: Original
1a: + Rounding during Train
1ab: + Noise
2: Train on DN
2a: + Account for Sparse Weights
2ab: + L1 Cost

91
900.0

2ab

10
Speedup Factor

Accuracy [%]

96

2 2a

1ab
1a

0.1

0.2

0.3

0.4
0.5
Threshold at Test

0.6

2
0.7

0.8

0.90

Fig. 4. Test accuracy results from standard GRUs run as delta networks after
training (curves 1, 1a, and 1ab) and those trained as delta networks (curves 2,
2a, and 2ab) under different constraints on the TIDIGITS dataset. The delta
networks are trained for Θ = 0.5. Note that the methods are combined, hence
the naming scheme. Additionally, the accuracy curve for 2 is hidden by the
curve 2a, since both achieve the same accuracy and only differ in speedup
metric.

Here the L1 norm is used to encourage sparse values in ∆h,
so that fewer delta updates are required.
VI.

R ESULTS

This section presents the results showing the trade-off between compute savings and accuracy loss from RNNs trained
on the TIDIGITS digit recognition benchmark. Furthermore,
it also demonstrates that the results found on small datasets
also appear in the much larger Wall Street Journal speech
recognition benchmark. The final example is for a CNN-RNN
stack trained on end-to-end steering control using a recent
driving dataset. The fixed-point Q3.4 (i.e m = 3 and f = 4)
format was used for network activation values in all speech
experiments except the “Original” RNN line for TIDIGITS in
Fig. 4, which was trained in floating-point representation. The
driving dataset in VI-C used Q2.5 activation. The networks
were trained with Lasagne [25] powered by Theano [26]. The
training time on an Nvidia GTX980 Ti GPU is reported to
indicate training difficulty, per discussions in the deep learning
symposium at NIPS 2016.
A. TIDIGITS dataset
The TIDIGITS dataset [21] was used as an initial evaluation task for the methods introduced in Sec. V. Single digits
(“oh” and zero through nine) from this database, with a total
of 2464 digits in the training set and 2486 digits in the test
set, were transformed in the standard way [27] to produce a
39-dimensional Mel-Frequency Cepstral Coefficient (MFCC)
feature vector using a 25 ms window, 10 ms frame shift, and
20 filter bank channels. The labels for “oh” and “zero” were
collapsed to a single label. Training time is approximately 8
minutes for 150 epochs of training per experiment.
The results of applying the methods introduced in Sec. V
can be found in Fig. 4. There are two quantities measured: the
change in the number of memory fetches, and the accuracy as a

92

900

1: Original
1a: + Rounding during Train
1ab: + Noise
2: Train on DN
2a: + Account for Sparse Weights
2ab: + L1 Cost

2

4

6

8
10
Speedup Factor [x]

12

14

16

Fig. 5. Accuracy-speedup tradeoff by adjusting threshold for TIDIGITS
dataset. By increasing the threshold (indicated by sample point size), greater
speedups can be obtained at greater losses of accuracy. For networks trained
as delta networks, the training threshold is the first (leftmost) point in the line
point sequence.

function of the threshold Θ. Fig. 5 shows the same results, but
removes the threshold axis to allow easier comparison among
the different training methods.
First, a standard GRU RNN was trained, achieving 96.59%
accuracy on the TIDIGITS task without data augmentation and
regularization. This network consists of a layer of 200 GRU
units connected to a layer of 200 fully-connected units and
finally to a classification layer for the 10 digits. This network
was then subsequently tested using the delta network GRU
formulation given in Sec. III.
The standard RNN run as a delta network (“Original”)
achieves 95% accuracy (a drop from zero delta threshold
accuracy of 96%) with a speedup factor of about 2.2X. That
is, only approximately 45% of the computes or fetches are
needed in achieving this accuracy. By adding the rounding
constraint during training (“+ Rounding during Training”), the
accuracy is nearly 97% with an increase to a 3.9X speedup.
By incorporating Gaussian noise (“+ Noise”), the accuracy can
be boosted even further to about 97% with a 4.2X speedup.
Essentially, these methods added generalization robustness
to the original GRU, while preventing small changes from
influencing the output of the network. These techniques allow
a higher threshold to be used while maintaining the same
accuracy, which results in a decrease of memory fetches and
a corresponding speedup.
Furthermore, training on the delta network itself (“Train on
DN”) allows a considerable speedup, achieving 98% accuracy
with a 8X speedup. Accounting for the effect of pre-existing
sparsity in the weight matrix (“+ Account for Sparse Weights”)
increases the speedup to 10X, without affecting the accuracy
(as it is the same network). Finally, incorporating an L1
cost on network changes in addition to training on the delta
network model (“+ L1 cost”) achieves 97% accuracy while
boosting speedup to 11.9X. Adding in the sparseness cost on
network changes decreases the accuracy slightly, since the loss
minimization must find a tradeoff between both error and delta

12

activation instead of considering error alone. However, using
the L1 loss can offer a significant additional speedup while
retaining an accuracy increase over the original GRU network.

10.3x

10
8.5x
8

7.3x
5.7x

6
4.2x

4
2

40

9.1x

30

Word Error Rate

Reduction in Ops [Multiples]

Finally, Fig. 5 also demonstrates the primary advantage
given by each algorithm; an increase in generalization robustness manifests as an overall upward shift in accuracy, while an
increase in sparsity manifests as a rightward shift in speedup.
Methods 1, 1a, and 1b generally increase generalization robustness while only modestly influencing the sparsity. Method 2
greatly increases both, while method 2a only increases sparsity,
and finally method 2ab slightly decreases accuracy but offers
the highest speedup.

50

Trained Delta Network
GRU Network used as a DN

6.2x

4.7x

20

3.1x

14.2%
12.1% 13.2%
11.0%
10.8%
10.4%
10.2% 10.2% 10.2%
1.3x

16.3%
10

B. Wall Street Journal dataset
While the gains seen on TIDIGITs are significant, the
delta network methodology was applied to an RNN trained
on a larger dataset to determine whether it could produce the
same gains. Here, the Wall Street Journal dataset comprised of
81 hours of transcribed speech, as described in [28]. Similar
to that study, the first 4 layers of the network consisted of
bidirectional GRU units with 320 units in each direction.
Training time for each experiment was about 120h.
Fig. 6 presents results on the achieved word error rate
(WER) and speedup on this dataset for two cases: First, running an existing speech transcription RNN as a delta network
(results shown as solid curves labeled “RNN used as a DN”),
and second, a network trained as a delta network with results
shown as the dashed curves “Trained Delta Network”. The
speedup here accounts for weight matrix sparsity as described
in Sec. V-D .
Surprisingly, the existing highly trained network already
shows significant speedup without loss of accuracy as the
threshold, Θ, is increased: At Θ = 0.2, the speedup is about
5.5 with a WER of 10.8% compared with the WER of 10.2%
at Θ = 0. However, training the RNN to run as a delta
network yields a network that achieves a slightly higher 5.7X
speedup with the same WER. For this large, multilayer RNN
that processes complex and constantly-changing speech data,
even the conventionally-trained RNN run as a delta network
can provide greater than 5X speedup with only a 5% increase
in the WER.
C. Comma.ai Driving DataSet
While speech applications are a common area of exploration for RNNs, driving scenarios are rapidly emerging as
another area of focused research. Here, the delta network
model was applied to determine the gains of exploiting the
redundancy of real-time video input. The open driving dataset
from comma.ai [29] with 7.25 hours of driving data was used,
with video data recorded at 20 FPS from a camera mounted on
the windshield. The network is trained to predict the steering
angle from the visual scene similar to [30], [31]. We followed
the approach in [30] by using an RNN on top of the CNN
feature detector as shown in Fig. 7. The CNN feature detector
has three convolution layers without pooling layers and a fullyconnected layer with 512 units. During training, the CNN
feature detector was pre-trained with an analog output unit
to learn the recorded steering angle from randomly selected
single frame images. Afterwards, the delta network RNN was

0

0.0

0.1

0.2
Threshold

0.3

0.4

0.50

Fig. 6. Accuracy and speedup tradeoffs on the Wall Street Journal dataset. The
solid lines show results from an existing deep RNN trained on this dataset
but run as a delta network. The dashed lines show results from a network
trained as a delta network using Θ = 0.2 during training. The horizontal
lines indicate the non-delta network accuracy level; similarly, the solid and
dashed horizontal lines indicate the accuracy of the normal network and the
DN network prior to rounding, respectively.

added, and was trained by feeding sequences of the visual
features from the CNN feature detector to learn sequences
of the steering angle. Since Q2.5 format was used for the
GRU layer activations, the GRU input and output vectors were
normalized to match this range.
However, this raw dataset results in a few practical difficulties and requires data preprocessing. In particular, the driver’s
intention of changing lanes or taking a specific route at the fork
of a road cannot be learned by training on this dataset, as no
intent or route goal is provided (though could be addressed
in the future using precise route planning and localization
techniques). Additionally, the ground truth of the steering
angle is noisy due to the sometimes unpredictable behavior
of the driver; for example, the driver occasionally idly adjusts
the steering wheel when the car is at a stop. As a a result,
the recorded steering angle occasionally becomes uncorrelated
with the direction of movement of the car. However, this issue
can be addressed in a straightforward way by excluding the
frames recorded during periods of low speed driving. Training
time of the CNN feature detector was about 8 hours for 10k
updates with the batch size of 200. Training of the RNN part
took about 3 hours for 5k updates with the batch size of 32
samples consisting of 48 frames/sample.
Fig. 8 shows the compute cost of the delta network GRU
layer in comparison with a conventional GRU, in the steering
angle prediction task on 2000 consecutive frames (100s) from
the validation set. While the number of operations per frame
remains constant for the conventional GRU layer, those for the
delta network GRU layer varies dynamically depending on the
change of visual features. Since the output of the CNN feature
detector is very stable over time as shown in the top figure,
a huge speedup of about 100X (see Fig. 9) is obtained by
removing the large temporal redundancy of the visual features
in driving data.
However, in this steering network, the computational cost

33

FC(1)

FC(1)

FC(100)

FC(100)

FC(1)

FC(100)

300
DN-GRU(512)

DN-GRU(512)
FC(512)

CNN

CNN

Conv(64K5S2)

CNN

324x

CNN only

200

Frame 2

Conv(16K8S4)

Frame 3

42x

3x320x160

Prediction error (degree)

Ops/frame

Visual feature unit

Fig. 7. Network architecture for steering angle prediction. The CNN feature
detector consists of three convolution layers (Conv) and a fully-connected
layer (FC) with 512 units. Conv(64K5S2) represents a convolution layer with
64 feature maps and 5x5 kernel with stride 2. Visual features of each image
frame are fed into the GRU-based RNN to predict steering angle.

0
100
200
300
400
500
10 7

Conventional GRU
DN GRU

10 6
10 5
10 4
10 3
60
40
20
0
20
40
60

CNN only
CNN + RNN
0

500

1000
Frame #

1500

2000

Fig. 8. Reduction of RNN compute cost in the steering angle prediction task
on the comma.ai driving dataset. The top figure shows the output of CNN
feature detector. The middle figure shows the required # of ops per frame
for the delta network GRU layer (trained with Θ = 0.1) in comparison with
the conventional GRU case. A huge speedup is obtained because of the large
temporal redundancy of the driving visual scenes. The bottom figure compares
the prediction errors of CNN predictor and CNN+RNN predictor. Note that
the RNN slightly improves the steering angle prediction by using multiple
frames. (See Fig. 9)

of the CNN (about 37 MOp/frame) dominates the RNN cost
(about 1.58 MOp/frame). Thus the overall, system-level computational savings for this example is only about 4.2%. However, future applications will likely have efficient dedicated
vision hardware or require a greater role for RNNs in processing numerous and complex data streams, which result in
RNN models that consume a greater percentage of the overall
energy/compute cost. Even now, steering angle prediction
already benefits from a delta network approach.
VII.

D ISCUSSION AND C ONCLUSION

Although the delta network concept can be applied to other
network architectures, as was shown in similar concurrent

30

177x

100
0
0.0

0.1

32
31

29

137x

Conv(32K5S2)
Frame 1

354x

253x

Speedup

DN-GRU(512)

405x

400

RMS Prediction err [degree]

(steering
angle scalar)

28

67x
0.2

Threshold

0.3

0.4

0.527

Fig. 9. Tradeoffs between prediction error and speedup of the GRU layer
on the steering angle prediction of the comma.ai driving dataset. The result
was obtained from 1000 samples with 48 consecutive frames sampled from
the validation set. Speedup here does not include weight matrix sparsity. The
network was trained with Θ = 0.1. A speedup of approximately 100X can
be obtained without increasing the prediction error, using Θ between 0.1 and
0.25.

work for CNNs [32], in practice a larger benefit is seen in
RNNs because we already need to store all the intermediate activation values for the delta networks. For example, the widelyused VGG19 CNN has 16M neuron states [19]. Employing the
delta network approach for CNNs requires doubled memory
access and a lot of additional memory space for neuron states.
Because the cost of external memory access is hundreds of
times larger than that of arithmetic operations, delta network
CNNs seem impractical without new memory technology to
address this issue.
In contrast, for RNNs, the number of weight parameters is
much bigger than the number of activations. The sparsity of the
deltas allows large savings in power consumption by reducing
the number of memory access for weight parameters. CNNs
do not have this advantage since the weight parameters are
shared by many units and their number is much smaller than
the number of activations. Whereas the work in [32] focuses on
an optimization method for converting a pre-trained CNN into
a Sigma-Delta network to reduce the compute cost, our work
shows that the delta networks can be optimized in terms of
accuracy and speedup by directly training the original network
to run as a delta network.
Recurrent neural networks can be highly optimized due to
the redundancy of their activations over time. When the use
of this temporal redundancy is combined with robust training
algorithms, this work demonstrates that speedups of 6X to
9X can be obtained with negligible accuracy loss in speech
RNNs, and speedups of over 100X are possible in steering
angle prediction RNNs.
Acknowledgements: We thank S. Braun for helping with the
WSJ speech transcription pipeline. This work was funded by
Samsung Institute of Advanced Technology, the University of
Zurich and ETH Zurich.

R EFERENCES

[21]

[1]

T. Mikolov, M. Karafiát, L. Burget, J. Cernockỳ, and S. Khudanpur,
“Recurrent neural network based language model.” in Interspeech,
vol. 2, 2010, p. 3.

[22]

[2]

D. Amodei, R. Anubhai, E. Battenberg, C. Case, J. Casper, B. Catanzaro, J. Chen, M. Chrzanowski, A. Coates, G. Diamos et al., “Deep
speech 2: End-to-end speech recognition in english and mandarin,”
arXiv preprint arXiv:1512.02595, 2015.

[23]

[3]

A. Graves, A.-r. Mohamed, and G. Hinton, “Speech recognition with
deep recurrent neural networks,” in Acoustics, Speech and Signal
Processing (ICASSP), 2013 IEEE International Conference on. IEEE,
2013, pp. 6645–6649.

[4]

L. Yao, A. Torabi, K. Cho, N. Ballas, C. Pal, H. Larochelle, and
A. Courville, “Describing videos by exploiting temporal structure,” in
Proceedings of the IEEE International Conference on Computer Vision,
2015, pp. 4507–4515.

[5]

K. Xu, J. Ba, R. Kiros, A. Courville, R. Salakhutdinov, R. Zemel, and
Y. Bengio, “Show, attend and tell: Neural image caption generation with
visual attention,” arXiv preprint arXiv:1502.03044, 2015.

[6]

S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural
computation, vol. 9, no. 8, pp. 1735–1780, 1997.

[7]

K. Cho, B. van Merrienboer, C. Gulcehre, F. Bougares, H. Schwenk,
and Y. Bengio, “Learning phrase representations using RNN
encoder-decoder for statistical machine translation,” arXiv preprint
arXiv:1406.1078, 2014.

[8]

M. Horowitz, “1.1 Computing’s energy problem (and what we can do
about it),” in 2014 IEEE International Solid-State Circuits Conference
Digest of Technical Papers (ISSCC), Feb. 2014, pp. 10–14.

[9]

M. Courbariaux, Y. Bengio, and J.-P. David, “Binaryconnect: Training
deep neural networks with binary weights during propagations,” in
Advances in Neural Information Processing Systems, 2015, pp. 3123–
3131.

[10]

E. Stromatias, D. Neil, M. Pfeiffer, F. Galluppi, S. B. Furber, and S.C. Liu, “Robustness of spiking Deep Belief Networks to noise and
reduced bit precision of neuro-inspired hardware platforms,” Frontiers
in Neuroscience, vol. 9, 2015.

[11]

M. Courbariaux and Y. Bengio, “Binarynet: Training deep neural
networks with weights and activations constrained to+ 1 or-1,” arXiv
preprint arXiv:1602.02830, 2016.

[12]

S. K. Esser, P. A. Merolla, J. V. Arthur, A. S. Cassidy, R. Appuswamy,
A. Andreopoulos, D. J. Berg, J. L. McKinstry, T. Melano, D. R. Barch
et al., “Convolutional networks for fast, energy-efficient neuromorphic
computing,” arXiv preprint arXiv:1603.08270, 2016.

[13]

M. Rastegari, V. Ordonez, J. Redmon, and A. Farhadi, “Xnor-net:
Imagenet classification using binary convolutional neural networks,”
arXiv preprint arXiv:1603.05279, 2016.

[14]

S. Han, H. Mao, and W. J. Dally, “Deep compression: Compressing
deep neural network with pruning, trained quantization and huffman
coding,” CoRR, abs/1510.00149, vol. 2, 2015.

[15]

S. Han, J. Kang, H. Mao, Y. Hu, X. Li, Y. Li, D. Xie, H. Luo,
S. Yao, Y. Wang et al., “Ese: Efficient speech recognition engine
with compressed lstm on fpga,” in FPGA 2017; NIPS 2016 EMDNN
workshop, 2016.

[16]

F. N. Iandola, M. W. Moskewicz, K. Ashraf, S. Han, W. J. Dally,
and K. Keutzer, “Squeezenet: Alexnet-level accuracy with 50x fewer
parameters and¡ 1mb model size,” arXiv preprint arXiv:1602.07360,
2016.

[17]

C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan,
V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,”
in Proceedings of the IEEE Conference on Computer Vision and Pattern
Recognition, 2015, pp. 1–9.

[18]

G. Huang, Y. Sun, Z. Liu, D. Sedra, and K. Weinberger, “Deep networks
with stochastic depth,” arXiv preprint arXiv:1603.09382, 2016.

[19]

K. Chatfield, K. Simonyan, A. Vedaldi, and A. Zisserman, “Return of
the devil in the details: Delving deep into convolutional nets,” arXiv
preprint arXiv:1405.3531, 2014.

[20]

M. Marszałek, I. Laptev, and C. Schmid, “Actions in context,” in IEEE
Conference on Computer Vision & Pattern Recognition, 2009.

[24]

[25]
[26]

[27]

[28]

[29]
[30]
[31]

[32]

R. G. Leonard and G. Doddington, “Tidigits speech corpus,” Texas
Instruments, Inc, 1993.
J. Chung, C. Gulcehre, K. Cho, and Y. Bengio, “Empirical evaluation of
gated recurrent neural networks on sequence modeling,” arXiv preprint
arXiv:1412.3555, 2014.
M. Courbariaux, Y. Bengio, and J.-P. David, “Low precision arithmetic
for deep learning,” arXiv preprint arXiv:1412.7024, 2014.
J. Ott, Z. Lin, Y. Zhang, S.-C. Liu, and Y. Bengio, “Recurrent
neural networks with limited numerical precision,” arXiv preprint
arXiv:1608.06902, 2016.
S. Dieleman et al., “Lasagne: First release.” Aug. 2015. [Online].
Available: http://dx.doi.org/10.5281/zenodo.27878
J. Bergstra, O. Breuleux, F. Bastien, P. Lamblin, R. Pascanu, G. Desjardins, J. Turian, D. Warde-Farley, and Y. Bengio, “Theano: a CPU
and GPU math expression compiler,” in Proceedings of the Python for
scientific computing conference (SciPy), vol. 4, 2010, p. 3.
D. Neil and S.-C. Liu, “Effective sensor fusion with event-based sensors
and deep network architectures,” in IEEE Int. Symposium on Circuits
and Systems (ISCAS), 2016.
S. Braun, D. Neil, and S.-C. Liu, “A curriculum learning method
for improved noise robustness in automatic speech recognition,” arXiv
preprint arXiv:1606.06864, 2016.
E. Santana and G. Hotz, “Learning a driving simulator,” arXiv preprint
arXiv:1608.01230, 2016.
M. Hempel, “Deep learning for piloted driving,” in NVIDIA GPU Tech
Conference, 2016.
M. Bojarski, D. D. Testa, D. Dworakowski, B. Firner, B. Flepp,
P. Goyal, L. D. Jackel, M. Monfort, U. Muller, J. Zhang, X. Zhang,
J. Zhao, and K. Ziebam, “End to end learning for self-driving cars,”
arXiv preprint arXiv:1604.07316, 2016.
P. O’Connor and M. Welling, “Sigma delta quantized networks,” arXiv
preprint arXiv:1611.02024, 2016.

