arXiv:1610.06160v1 [cs.LG] 19 Oct 2016

CBMM Memo No. 057

October 19, 2016

Streaming Normalization: Towards Simpler and
More Biologically-plausible Normalizations for
Online and Recurrent Learning
by
Qianli Liao1,2 , Kenji Kawaguchi2 and Tomaso Poggio1,2
1. Center for Brains, Minds and Machines, McGovern Institute
2. Computer Science and Artificial Intelligence Laboratory
MIT

Abstract: We systematically explored a spectrum of normalization algorithms related to Batch Normalization
(BN) and propose a generalized formulation that simultaneously solves two major limitations of BN: (1) online
learning and (2) recurrent learning. Our proposal is simpler and more biologically-plausible. Unlike previous
approaches, our technique can be applied out of the box to all learning scenarios (e.g., online learning, batch
learning, fully-connected, convolutional, feedforward, recurrent and mixed — recurrent and convolutional) and
compare favorably with existing approaches. We also propose Lp Normalization for normalizing by different orders
of statistical moments. In particular, L1 normalization is well-performing, simple to implement, fast to compute,
more biologically-plausible and thus ideal for GPU or hardware implementations.

This work was supported by the Center for Brains, Minds and Machines
(CBMM), funded by NSF STC award CCF - 1231216.

1

Approach

FF & FC

FF & Conv

Rec & FC

Rec & Conv

Online
Learning

Small
Batch

All
Combined

3

3

7

7

7

Suboptimal

7

3

3

Limited

Limited

7

Suboptimal

7

3

7*

3

7*

3

3

7*

3

3

3

3

3

3

3

Original Batch
Normalization(BN)
Time-specific BN
Layer
Normalization
Streaming
Normalization

Table 1: An overview of normalization techiques for different tasks. 3: works well. 7: does not work well. FF:
Feedforward. Rec: Recurrent. FC: Fully-connected. Conv: convolutional. Limited: time-specific BN requires
recording normalization statistics for each timestep and thus may not generalize to novel sequence length. *Layer
normalization does not fail on these tasks but perform significantly worse than the best approaches.

1

Introduction

Batch Normalization [Ioffe and Szegedy, 2015] (BN) is a highly effective technique for speeding up convergence
of feedforward neural networks. It enabled recent development of ultra-deep networks [He et al., 2015] and some
biologically-plausible variants of backpropagation [Liao et al., 2015]. However, despite its success, there are two
major learning scenarios that cannot be handled by BN: (1) online learning and (2) recurrent learning.
For the second scenario recurrent learning, [Liao and Poggio, 2016] and [Cooijmans et al., 2016] independently proposed “time-specific batch normalization”: different normalization statistics are used for different timesteps of an RNN.
Although this approach works well in many experiments in [Liao and Poggio, 2016] and [Cooijmans et al., 2016], it
is far from perfect due to the following reasons: First, it does not work with small mini-batch or online learning.
This is similar to the original batch normalization, where enough samples are needed to compute good estimates of
statistical moments. Second, it requires sufficient training samples for every timestep of an RNN. It is not clear how
to generalize the model to unseen timesteps. Finally, it is not biologically-plausible. While homeostatic plasticity
mechanisms (e.g., Synaptic Scaling) [Turrigiano and Nelson, 2004, Stellwagen and Malenka, 2006, Turrigiano, 2008]
are good biological candidates for BN, it is hard to imagine how such normalizations can behave differently for
each timestep. Recently, Layer Normalization (LN) [Ba et al., 2016] was introduced to solve some of these issues. It
performs well in feedforward and recurrent settings when only fully-connected layers are used. However, it does not
work well with convolutional networks. A summary of normalization approaches in various learning scenarios is
shown in Table 1.
We note that different normalization methods like BN and LN can be described in the same framework detailed
in Section 3. This framework introduces Sample Normalization and General Batch Normalization (GBN) as
generalizations of LN and BN, respectively. As their names imply, they either collect normalization statistics from a
single sample or from a mini-batch. We explored many variants of these models in the experiment section.
A natural and biologically-inspired extension of these methods would be Streaming Normalization: normalization
statistics are collected in an online fashion from all previously seen training samples (and all timesteps if recurrent).
We found numerous advantages associated with this approach: 1. it naturally supports pure online learning or
learning with small mini-batches. 2. for recurrent learning, it is more biologically-plausible since a unique set of
normalization statistics is maintained for all timesteps. 3. it performs well out of the box in all learning scenarios
(e.g., online learning, batch learning, fully-connected, convolutional, feedforward, recurrent and mixed — recurrent
and convolutional). 4. it offers a new direction of designing normalization algorithms, since the idea of maintaining
online estimates of normalization statistics is independent from other design choices, and as a result any existing
algorithm (e.g., in Figure 1 C and D) can be extended to a “streaming” setting.
We also propose Lp normalization: instead of normalizing by the second moment like BN, one can normalize by the
p-th root of the p-th absolute moment (See Section 3.4 for details). In particular, L1 normalization works as well as
the conventional approach (i.e., L2) in almost all learning scenarios. Furthermore, L1 normalization is easier to
implement: it is simply the average of absolute values. We believe it can be used to simplify and speed up BN and
our Streaming Normalization in GPGPU, dedicated hardware or embedded systems. L1 normalization may also

2

be more biologically-plausible since the gradient of the absolute value is trivial to implement, even for biological
neurons.
In the following section, we introduce a simple training scheme, a minor but necessary component of our formulation.

2

Online and Batch Learning with “Decoupled Accumulation and Update”

Although it is not our main result, we discuss a simple but to the best of our knowledge less explored1 training
scheme we call a “Decoupled Accumulation and Update” (DAU).
Conventionally, the weights of a neural network are updated every mini-batch. So the accumulation of gradients
and weights update are coupled. We note that a general formulation would be that while for every mini-batch the
gradients are still accumulated, one does not necessarily update the weights. Instead, the weights are updated every
n mini-batches. The gradients are cleared after each weight update. Two parameters characterize this procedure:
Samples per Batch (S/B) m and Batch per Update (B/U) n. In conventional training, n = 1.
Note that this procedure is similar to but distinct from simply training larger mini-batches since every
mini-batch arrives in a purely online fashion so one cannot look back into any previous mini-batches. For example,
if batch normalization is present, performing this procedure with m S/B and n B/U is different from that with
m ∗ n S/B and 1 B/U.
If m = 1, it reduces to a pure online setting where one sample arrives at a time. The key advantage of this proposal
over conventional training is that we explicitly require less frequent (but more robust) weight updates. The memory
requirement (in additional to storing the network weights) is storing m samples and related activations.
This training scheme generalizes the conventional approach, and we found that merely applying this approach
greatly mitigated (although not completely solved) the catastrophic failure of training batch normalization with
small mini-batches (See Figure 4). Therefore, we will use this formulation throughout our paper.
We also expect this scheme to benefit learning sequential recurrent networks with varying input sequence lengths.
Sometimes it is more efficient to pack in a mini-batch training samples with the same sequence length. If this is the
case, our approach predicts that it would be desirable to process multiple such mini-batches with varying sequence
lengths before a weight update. Accumulating gradients from training different sequence lengths should provide a
more robust update that works for different sequence lengths, thus better approximating the true gradient of the
dataset.
In Streaming Normalization with recurrent networks, it is often beneficial to learn with n > 1 (i.e., more than one
batch per update). The first mini-batch collects all the normalization statistics from all timesteps so that later
mini-batches are normalized in a more stable way.

3

A General Framework for Normalization

We propose a general framework to describe different normalization algorithms. A normalization can be thought of as
a process of modifying the activation of a neuron using some statistics collected from some reference activations. We
adopt three terms to characterize this process: A Normalization Operation (NormOP) is a function N (xi , si )
that is applied to each neuron i to modify its value from xi to N (xi , si ), where si is the Normalization Statistics
(NormStats) for this neuron. NormStats is any data required to perform NormOP, collected using some function
si = S(Ri ), where Ri is a set of activations (could include xi ) called Normalization Reference (NormRef ).
Different neurons may or may not share NormRef and NormStats.
1 We

are not aware of this approach in the literature. If there is, please inform us.

3

This framework captures many previous normalizations algorithms as special cases. For example, the original Batch
Normalization (BN) [Ioffe and Szegedy, 2015] can be described as follows: the NormRef for each neuron i is all
activations in the same channel of the same layer and same batch (See Figure 1 C for illustration). The NormStats
are Si ={µi ,σi } — the mean and standard deviation of the activations in NormRef. The NormOp is N(xi ,{µi ,σi })
i
= xiσ−µ
i
For the recent Layer Normalization [Ba et al., 2016], NormRef is all activations in the same layer (See Figure 1 C).
The NormStats and NormOp are the same as BN.
We group normalization algorithms into three categories:
1. Sample Normalization (Figure 1 C): NormStats are collected from one sample.
2. General Batch Normalization (Figure 1 D): NormStats are collected from all samples in a mini-batch.
3. Streaming Normalization: NormStats are collected in an online fashion from all pass training samples.
In the following sections, we detail each algorithm and provide pseudocode.

3.1

Sample Normalization

Sample normalization is the simplest category among the three. NormStats are collected only using the activations
in the current layer of current sample. The computation is the same at training and test times. It handles online
learning naturally. Layer Normalization [Ba et al., 2016] is an example of this category. More examples are shown
in Figure 1 C.
The pseudocode of forward and backpropagation is show in Algorithm 1 and Algorithm 2.
Algorithm 1 Sample Normalization Layer: Forward
Require: layer input x (a sample), NormOP N(.,.), function S(.) to compute NormStats for every element of x
Ensure: layer output y (a sample)
s = S(x)
y = N(x,s)

Algorithm 2 Sample Normalization Layer: Backpropagation
Require: ∂E
∂y (a sample) where E is objective, layer input x, NormOP N(.,.), function S(.) to compute NormStats
for every element of x
Ensure: ∂E
∂x (a sample)
∂E
can
be
calculated using chain rule. Detail omitted.
∂x

3.2

General Batch Normalization

In General Batch Normalization (GBN), NormStats are collected in some way using the activations from all
samples in a training mini-batch. Note that one cannot really compute the batch NormStats at test time since test
samples should be handled independently from each other, instead in a batch. To overcome this, one can simply
compute running estimates of training NormStats and use them for testing (e.g., the original Batch Normalization
[Ioffe and Szegedy, 2015] computes moving averages).
More examples of GBN are shown in Figure 1 D. The pseudocode is shown in Algorithm 3 and 4.

4

(A)

(B) Decoupled Accumulation and Update
Mini-batch (size m>=1)
X
Y

X

Feature
Maps

A layer in the network

Y

X

X
Feature
Maps

Y

...

X

Y

Feature
Maps

Feature
Maps

Y

Feature
Maps

X
Y

X
Feature
Maps

Y

Feature
Maps

Update weights every n (>=1) mini-batches
X
Y

X

X
Feature
Maps

Y

...

X

Y

Feature
Maps

Feature
Maps

Y

Feature
Maps

X
Y

X
Feature
Maps

Y

Feature
Maps

(C) Sample Normalization
SP1. Across x, y
X
Y

SP2. Layer Normalization (across layer)

Mini-batch
Feature
Maps

Sample 1

X

Y

X

X

Y

Feature
Maps

X

Y

Feature
Maps

Feature
Maps

Y

Feature
Maps

Sample 2

SP4. Across local x, y

SP5. Across features
X

X
Y

X

Y

Feature
Maps

SP3. Across features and local x,y

Feature
Maps

X

Y

X

Y

Feature
Maps

Feature
Maps

Y

Feature
Maps

Blue indicates the
range to collect
normalization statistics

(D) General Batch Normalization
BA1. Batch Normalization
(across x, y and samples in a mini-batch)
X
Y

Mini-batch
Feature
Maps

Sample 1

X

X
Y

Y

X
Feature
Maps

Feature
Maps

Y

X

X
Y

Feature
Maps

Feature
Maps

Y

Feature
Maps

Sample 2

BA4. Across local x, y and samples

Feature
Maps

Y

Y
Feature
Maps

BA6.Neuron-wise Normalization (across samples)

BA5. Across features and samples
X

X

X
Y

BA3. Across features, local x,y and samples

BA2. Across layer and samples

X
Feature
Maps

Y

X
Feature
Maps

X

Y

Feature
Maps

Y

Feature
Maps

(E) Streaming Normalization
Previous Mini-batches

...

Current mini-batch

...

Weight
Update

Any conﬁguration in (C) or (D) can be made streaming.

Figure 1: A General Framework of Normalization. A: the input to convolutional layer is a 3D matrix consists of 3

dimensions: x (image width), y (image height) and features/channel. For fully-connected layers, x=y=1. B: training
with decoupled accumulation and update. C: Sample Normalization. D: General Batch Normalization. E: Streaming
Normalization

5

Algorithm 3 General Batch Normalization Layer: Forward
Require: layer input x (a mini-batch), NormOP N(.,.), function S(.) to compute NormStats for every element of
x, running estimates of NormStats ŝ, function F to update ŝ.
Ensure: layer output y (a mini-batch), running estimates of NormStats ŝ
if training then
s = S(x)
ŝ = F(ŝ,s)
y = N(x,s)
else {testing}
y = N(x,ŝ)
end if
Algorithm 4 General Batch Normalization Layer: Backpropagation
Require: ∂E
∂y (a mini-batch) where E is objective, layer input x (a mini-batch), NormOP N(.,.), function S(.) to
compute NormStats for every element of x, running estimates of NormStats ŝ, function F to update ŝ.
Ensure: ∂E
∂x (a mini-batch)
∂E
can
be
calculated using chain rule. Detail omitted.
∂x

3.3

Streaming Normalization

Finally, we present the main results of this paper. We propose Streaming Normalization: NormStats are collected in
an online fashion from all past training samples. The main challenge of this approach is that it introduces infinitely
long dependencies throughout the neuron’s activation history — every neuron’s current activation depends on all
previously seen training samples.
It is intractable to perform exact backpropagation on this dependency graph for the following reasons: there is
no point backpropagating beyond the last weight update (if any) since one cannot redo the weight update. This,
on the other hand, would imply that one cannot update the weights until having seen all future samples. Even
backpropagating within a weight update (i.e., the interval between two weight updates, consisting of n mini-batches)
turns out to be problematic: one is usually not allowed to backpropagate to the previous several mini-batches since
they are discarded in many practical settings.
For the above reasons, we abandoned the idea of performing exact backpropagation for streaming normalization.
Instead, we propose two simple heuristics: Streaming NormStats and Streaming Gradients. They are discussed
below and the pseudocode is shown in Algorithm 5 and Algorithm 6.

3.3.1

Streaming NormStats

Streaming NormStats is a natural requirement of Streaming Normalization, since NormStats are collected from all
previously seen training samples. We maintain a structure/table H1 to keep all the information needed to generate
a good estimate of NormStats. and update it using a function F everytime we encounter a new training sample.
Function F also generates the current estimate of NormStats called ŝ. We use ŝ to normalize instead of s. See
Algorithm 5 for details.
There could be many potential designs for F and H1 , and in our experiments we explored a particular version:
we compute two sets of running estimates to keep track of the long-term and short-term NormStats: H1 =
{ŝlong , ŝshort , counter}.
• Short-term NormStats ŝshort is the exact average of NormStats s since the last weight update. The
counter keeps track of the number of times a different s is encountered to compute the exact average of s.
• Long-term NormStats ŝlong is an exponential average of ŝshort since the beginning of training.
6

Whenever the weights of the network is updated: ŝlong = κ1 ∗ ŝlong + κ2 ∗ ŝshort , counter is reset to 0 and ŝshort is
set to empty. In our experiments, κ1 + κ2 = 1 so that an exponential average of ŝshort is maintained in ŝlong .
In our implementation, before testing the model, the last weight update is NOT performed, and ŝshort is also NOT
cleared, since ŝshort is needed for testing 2
In addition to updating H1 in the way described above, the function F also computes ŝ = α1 ŝlong + α2 ŝshort . Finally,
ŝ is used for normalization.

3.3.2

Streaming Gradients

We maintain a structure/table H2 to keep all the information needed to generate a good estimate of gradients
of NormStats and update it using a function G everytime backpropagation reaches this layer. Function G also
c
E
∂
E
generates the current estimate of gradients of NormStats called ∂c
∂ ŝ . We use ∂ ŝ for further backpropagation instead
∂E
of ∂ ŝ . See Algorithm 6 for details.
Again, there could be many potential designs for G and H2 , and we explored a particular version: we compute
two sets of running estimates to keep track of the long-term and short-term gradients of NormStats: H2 =
{ĝlong , ĝshort , counter}.
• Short-term Gradients of NormStats ĝshort is the exact average of gradients of NormStats ∂E
∂ ŝ since the last
∂E
weight update. The counter keeps track of the number of times a different ∂ ŝ is encountered to compute
the exact average of ∂E
∂ ŝ .
• Long-term Gradients of NormStats ĝlong is an exponential average of ĝshort since the beginning of
training.

Whenever the weights of network is updated: ĝlong = κ3 ∗ ĝlong + κ4 ∗ ĝshort , counter is reset to 0 and ĝshort is set
to empty. In our experiments, κ3 + κ4 = 1 so that an exponential average of ĝshort is maintained in ĝlong .
∂E
E
In addition to updating H2 in the way described above, the function G also computes ∂c
∂ ŝ = β1 ĝlong +β2 ĝshort +β3 ∂ ŝ .
E
Finally, ∂c
is used for further backpropagation.
∂ ŝ

Algorithm 5 Streaming Normalization Layer: Forward
Require: layer input x (a mini-batch), NormOP N(.,.), function S(.) to compute NormStats for every element
of x, running estimates of NormStats and/or related information packed in a structure/table H1 , function F to
update H1 and generate current estimate of NormStats ŝ.
Ensure: layer output y (a mini-batch) and H1 (it is stored in this layer, instead of feeding to other layers), always
maintain the latest ŝ in case of testing
if training then
s = S(x)
{H1 ,ŝ} = F(H1 ,s)
y = N(x,ŝ)
else {testing}
y = N(x,ŝ)
end if
2 It also possible to simply store the last ŝ computed in training for testing, instead of storing ŝ
short and re-computing ŝ in testing.
These two options are mathematically equivalent.

7

Algorithm 6 Streaming Normalization Layer: Backpropagation
Require: ∂E
∂y (a mini-batch) where E is objective, layer input x (a mini-batch), NormOP N(.,.), function S(.) to
compute NormStats for every element of x, running estimates of NormStats ŝ, running estimates of gradients
and/or related information packed in a structure/table H2 , function G to update H2 and generate the current
E
estimates of gradients of NormStats ∂c
.
∂ ŝ

Ensure: ∂E
∂x (a mini-batch) and H2 (it is stored in this layer, instead of feeding to other layers)
∂E
∂ ŝ is calculated using chain rule.
E
} = G(H , ∂E )
{H , ∂c
2 ∂ ŝ
2 ∂ ŝ
c
∂
E
Use ∂ ŝ for further backpropagation,
∂E
∂x is calculated using chain rule.

3.3.3

instead of

∂E
∂ ŝ

A Summary of Streaming Normalization Design and Hyperparameters

i
The NormOp used in this paper is N(xi ,{µi ,σi }) = xiσ−µ
, the same as what is used by BN. The NormStats are
i
collected using one of the Lp normalization schemes described in Section 3.4.

With our particular choices of F , G, H1 and H2 , the following hyperparameters uniquely characterize a Streaming
Normalization algorithm: α1 , α2 , β1 , β2 , β3 , κ1 , κ2 , κ3 , κ4 , samples per batch m, batches per update n and a choice
of mini-batch NormRef (i.e., SP1-SP5, BA1-BA6 in Figure 1 C and D).
Unless mentioned otherwise, we use BA1 in Figure 1 D as the NormRef throughout the paper. We also demonstrate
the use of other NormRefs (BA4 and BA6) in the Appendix Figure A1.
An important special case: If n = 1, α1 = 0, α2 = 1, β1 = 0, β2 = 0, β3 = 1, we ignore all NormStats and gradients
beyond the current mini-batch. The algorithm reduces to exactly the GBN algorithm. So Streaming
Normalization is strictly a generalization of GBN and thus also captures the original BN [Ioffe and Szegedy, 2015]
as a special case. Recall from Section 3.3.1 that ŝshort is NOT cleared before testing the model. Thus for testing,
NormStats are inherited from the last training mini-batch. It works well in practice.
Unless mentioned otherwise, we set α1 = κ1 = κ3 , α2 = κ2 = κ4 , κ1 + κ2 = 1, κ3 + κ4 = 1, α1 + α2 = 1,
β1 + β2 + β3 = 1. We leave it to future research to explore different choices of hyperparameters (and perhaps other
F and G).

3.3.4

Implementation Notes

One minor drawback of not performing exact backpropagation is that it may break the gradient check of the entire
model. One solution coulde be: (1) perform gradient check of the model without SN and then add a correctly
implemented SN. (2) make sure the SN layer is correctly coded (e.g., by reducing it to standard BN using the
hyperparameters discussed above and then perform gradient check).

3.4

Lp Normalization: Calculating NormStats with Different Orders of Moments

Let us discuss the function S(.) for calculating NormStats. There are several choices for this function. We propose
Lp normalization. It captures the previous mean-and-standard-deviation normalization as a special case.
First, mean µ is always calculated the same way — the average of the activations in NormRef. The divisive factor
σ, however, can be calculated in several different ways. In Lp Normalization, σ is chosen to be the p-th root of
the p-th Absolute Moment.

8

Here the Absolute Moment of a distribution P (x) about a point c is:
Z
and the discrete form is:

|x − c|p P (x)dx
N
1 X
|xi − c|p
N i=1

(1)

(2)

Lp Normalization can be performed with three settings:
• Setting A: σ is the p-th root of the p-th absolute moment of all activations in NormRef with c being the
mean µ of NormRef.
• Setting B: σ is the p-th root of the p-th absolute moment of all activations in NormRef with c being the
running estimate µ̂ of the average.
• Setting C: σ is the p-th root of the p-th absolute moment of all activations in NormRef with c being 0.
Most of these variants have similar performance but some are better in some situations.
We call it Lp normalization since it is similar to the norm in the Lp space.
Setting B and C are better for online learning since A will give degenerate result (i.e., σ = 0) when there is only one
sample in NormRef. Empirically, when there are enough samples in a mini-batch, A and B perform similarly.
We discuss several important special cases:
Special Case A-2: setting A with n=2, σ is the standard deviation (square root of the 2nd moment)
of all activations in NormRef. This setting is what is used by Batch Normalization [Ioffe and Szegedy, 2015]
and Layer Normalization [Ba et al., 2016].
Special Case p=1: Whenever p=1, σ is simply the average of absolute values of activations. This setting
works virtually the same as p=2, but is much simpler to implement and faster to run. It might also be more
biologically-plausible, since the gradient computations are much simpler.

3.5

Separate Learnable Bias and Gain Parameters

The original Batch Normalization [Ioffe and Szegedy, 2015] also learns a bias and a gain (i.e., shift and scaling)
parameter for each feature map. Although not usually done, but clearly these shift and scaling operations can
be completely separated from the normalization layer. We implemented them as a separate layer following each
normalization layer. These parameters are learned in the same way for all normalization schemes evaluated in this
paper.

4

Generalization to Recurrent Learning

In this section, we generalize Sample Normalization, General Batch Normalization and Streaming Normalization to
recurrent learning. The difference between recurrent learning and feedforward learning is that for each training
sample, every hidden layer of the network receives t activations h1 , , ..., ht , instead of only one.

9

4.1

Recurrent Sample Normalization

Sample Normalization naturally generalizes to recurrent learning since all NormStats are collected from the current
layer of the current timestep. Training and testing algorithms remain the same.

4.2

Recurrent General Batch Normalization (RGBN)

The generalization of GBN to recurrent learning is the same as what was proposed by [Liao and Poggio, 2016] and
[Cooijmans et al., 2016]. The training procedure is exactly the same as before (Algorithm 3 and Algorithm 4). For
testing, one set of running estimates of NormStats is maintained for each timestep.
Another way of viewing this algorithm is that the same GBN layers described in Algorithm 3 and 4 are used in
the unrolled recurrent network. Each GBN layer in the unrolled network uses a different memory storage for
NormStats.

4.3

Recurrent Streaming Normalization

Extending Streaming Normalization to recurrent learning is straightforward – we not only stream through all the
past samples, but also through all past timesteps. Thus, we maintain a unique set of running estimates of NormStats
for all timesteps. This is more biologically-plausible and memory efficient than the above approach (RGBN).
Again, another way of viewing this algorithm is that the same Streaming Normalization layers described in Algorithm
5 and 6 are used in the original (instead of the unrolled) recurrent network. All unrolled versions of the same
layer share running estimates of NormStats and other related data.
One caveat is that as time proceeds, the running estimates of NormStats are slightly modified. Thus when
backpropagation reaches the same layer again, the NormStats are slightly different from the ones originally used
for normalization. Empirically, it seems to not cause any problem on the performance. Training with “decoupled
accumulation update” with Batches per Update (B/U) > 1 3 can also mitigate this problem, since it makes NormStats
more stable over time.

5

Streaming Normalized RNN and GRU

In our character-level language modeling task, we tried Normalized Recurrent Neural Network (RNN) and Normalized
Gated Recurrent Unit (GRU) [Chung et al., 2014]. Let us use Norm(.) to denote a normalization, which can be
either Sample Normalization, General Batch Normalization or Streaming Normalization. A bias and gain parameter
is also learned for each neuron. We use NonLinear to denote a nonlinear function. We used hyperbolic tangent
(tanh) nonlinearity in our experiments. But we observed ReLU also works. ht is the hidden activation at time t. xt
is the network input at time t. W. denotes the weights.
denotes elementwise multiplication.
Normalized RNN
ht = N onLinear(N orm(Wx ∗ xt ) + N orm(Wh ∗ ht−1 ))

(3)

Normalized GRU
gr = Sigmoid(N orm(Wxr ∗ xt ) + N orm(Whr ∗ ht−1 )))

(4)

gz = Sigmoid(N orm(Wxz ∗ xt ) + N orm(Whz ∗ ht−1 )))

(5)

hnew = N onLinear(N orm(Wxh ∗ xt ) + N orm(Whh ∗ (ht−1

gr )))

(6)

hnew + (1 − gz )

ht−1

(7)

ht = gz
3 B/U=2

is often enough

10

6

Related Work

[Laurent et al., 2015] and [Amodei et al., 2015] used Batch Normalization (BN) in stacked recurrent networks,
where BN was only applied to the feedforward part (i.e., “vertical” connections, input to each RNN), but not the
recurrent part (i.e., “horizontal”, hidden-to-hidden connections between timesteps). [Liao and Poggio, 2016] and
[Cooijmans et al., 2016] independently proposed applying BN in recurrent/hidden-to-hidden connections of recurrent
networks, but separate normalization statistics must be maintained for each timestep. [Liao and Poggio, 2016]
demonstrated this idea with deep multi-stage fully recurrent (and convolutional) neural networks with ReLU
nonlinearities and residual connections. [Cooijmans et al., 2016] demonstrated this idea with LSTMs on language
processing tasks and sequential MNIST. [Ba et al., 2016] proposed Layer Normalization (LN) as a simple normalization technique for online and recurrent learning. But they observed that LN does not work well with
convolutional networks. [Salimans and Kingma, 2016] and [Neyshabur et al., 2015] studied normalization using
weight reparameterizations. An early work by [Ullman and Schechtman, 1982] mathematically analyzed a form of
online normalization for visual perception and adaptation.

7
7.1

Experiments
CIFAR-10 architectures and Settings

We evaluated the normalization techniques on CIFAR-10 dataset using feedforward fully-connected networks, feedforward convolutional network and a class of convolutional recurrent networks proposed by [Liao and Poggio, 2016].
The architectural details are shown in Figure 2. We train all models with learning rate 0.1 for 25 epochs and 0.01
for 5 epochs. Momentum 0.9 is used. We used MatConvNet [Vedaldi and Lenc, 2015] to implement our models.
1x1x10

Output
FC

1x1x128

1x1x10
Norm-AvgPool

1x1x10

16x16x128

Output
FC

1x1x10

16x16x128

1x1x256

1x1x256

Norm-ReLU-Conv * 2

Identity
32x32x32

Input

(A) FC & FeedForward

runs k2 times with shared weights
Norm-ReLU-Conv * 2 + Idenity
Norm-ReLU-Conv
runs k1 times with shared weights
Norm-ReLU-Conv * 2 + Idenity

32x32x32
Conv

32x32x3

8x8x256
Norm-AvgPool

16x16x128

FC-Norm-ReLU
32x32x3

Norm-AvgPool

1x1x128

32x32x32
FC-Norm-ReLU

1x1x128

Output
FC

Norm-ReLU-Conv

FC-Norm-ReLU
1x1x256

FC

Norm-ReLU-Conv * 2

Identity

Input

(B) Conv. & Feedforward

Output

Input

(C) Conv. & Recurrent

Norm-ReLU-Conv * 2

16x16x128

Norm-ReLU-Conv * 2 + Idenity

Norm-ReLU-Deconv * 2

Norm-ReLU-Conv * 2
Norm-ReLU-Conv * 2 + Idenity

32x32x64

Conv

32x32x3

Norm-ReLU-Conv * 2 + Idenity

Norm-ReLU-Deconv * 2

Conv

32x32x3

Input

All transitions within
the box run k times
concurrently with
weights shared
over time

(D) Conv. & Densely Recurrent

Figure 2: Architectures for CIFAR-10. Note that C reduces to B when k1 = k2 = 1.

7.2

Lp Normalization

We show BN with Lp normalization in Figure 3. Note that Lp normalization can be applied to Layer Normalization
and all other normalizations show in 1 C and D. L1 normalization works as well as L2 while being simpler to
implement and faster to compute.

11

40

40
L7 BN
L5 BN
L4 BN
L3 BN
L2 BN (Original)
L1 BN

35

35

30

Validation Error on CIFAR-10

Training Error on CIFAR-10

30

25

20

25

20

15

15

10

10

5

L7 BN
L5 BN
L4 BN
L3 BN
L2 BN (Original)
L1 BN

0

5

10

15

20

25

5

30

0

5

10

Epoch

15

20

25

30

Epoch

Figure 3: Lp Normalization. The architecture is a feedforward and convolutional network (shown in Figure 2

B). All statistical moments perform similarly well. L7 normalization is slightly worse.

7.3

Online Learning or Learning with Very Small Mini-batches

We perform online learning or learning with small mini-batches using architecture A in Figure 2.
Plain Mini-batch vs. Decoupled Accumulation and Update (DAU): We show in Figure 4 comparisons
between conventional mini-batch training and Decoupled Accumulation and Update (DAU).

12

90

90
BN, 10 S/B, 1 B/U
BN, 5 S/B, 40 B/U
BN, 10 S/B, 20 B/U
BN, 200 S/B, 1 B/U

85

80

Validation Error on CIFAR-10

Training Error on CIFAR-10

80

BN, 10 S/B, 1 B/U
BN, 5 S/B, 40 B/U
BN, 10 S/B, 20 B/U
BN, 200 S/B, 1 B/U

85

75
70
65
60
55
50
45
40

75
70
65
60
55
50
45
40

35

35
0

10

20

30

0

10

20

Epoch

30

Epoch

Figure 4: Plain Mini-batch vs. Decoupled Accumulation and Update (DAU). The architecture is a feedforward
and fully-connected network (shown in Figure 2 A). S/B: Samples per Batch. B/U: Batches per Weight Update.
We show there are significant performance differences between plain mini-batch (i.e., B/U=1) and Decoupled
Accumulation and Update (DAU, i.e., B/U=n>1). DAU significantly improves the performance of BN with small
number of samples per mini-batch (e.g., compare curve 1 with 3).

Layer Normalization vs. Batch Normalization vs. Streaming Normalization: We compare in Figure 5
Layer Normalization, Batch Normalization and Streaming Normalization with different choices of S/B and B/U.
50

Validation Error on CIFAR-10

70

BN, 5 S/B, 1 B/U
BN, 5 S/B, 40 B/U
LN, 200 S/B, 1 B/U
Ours, 200 S/B, 1 B/U
Ours, 2 S/B, 100 B/U
Ours, 1 S/B, 200 B/U
BN, 200 S/B, 1 B/U

80

48

48

60

50

70

60

46

44

42
BN, 5 S/B, 1 B/U
BN, 5 S/B, 40 B/U
LN, 200 S/B, 1 B/U
Ours, 200 S/B, 1 B/U
Ours, 2 S/B, 100 B/U
Ours, 1 S/B, 200 B/U
BN, 200 S/B, 1 B/U

50
40

40

40

0

10

20

Epoch

30

0

10

20

38

30

Epoch

Validation Error on CIFAR-10

BN, 5 S/B, 1 B/U
BN, 5 S/B, 40 B/U
LN, 200 S/B, 1 B/U
Ours, 200 S/B, 1 B/U
Ours, 2 S/B, 100 B/U
Ours, 1 S/B, 200 B/U
BN, 200 S/B, 1 B/U

80

Training Error on CIFAR-10

50
90

Training Error on CIFAR-10

90

5

10

15

20

Epoch

46

44

42
BN, 5 S/B, 1 B/U
BN, 5 S/B, 40 B/U
LN, 200 S/B, 1 B/U
Ours, 200 S/B, 1 B/U
Ours, 2 S/B, 100 B/U
Ours, 1 S/B, 200 B/U
BN, 200 S/B, 1 B/U

40

25

30

38

0

10

20

30

Epoch

Figure 5: Different normalizations applied to a feedforward and fully-connected network (shown in Figure 2 A).
The right two pannels are zoomed-in versions of the left two pannels. S/B: Samples per Batch. B/U: Batches
per Weight Update. “Ours” refers to Streaming Normalization with “L1 norm” (Setting B with p=1 in Section 3.4)
and α1 = β1 = 0.7, α2 = β2 = 0.3 and β3 = 0 (see Section 3.3.3 for more details about hyperparameters). We show
that our algorithm works with pure online learning (1 S/B) and tiny mini-batch (2 S/B), and it outperforms Layer
Normalization. The choice of S/B does not matter for layer normalization since it processes samples independently.

13

7.4

Evaluating Variants of Batch Normalization

Feedforward Convolutional Networks: In Figure 6, we tested algorithms shown in Figure 1 C and D using the
architecture B in Figure 2. We also show the performance of our Streaming Normalization for reference.
50
SP1: XY, sample
SP2: layer, sample (LN)
SP3: localXY, feat, sample
SP4: localXY, sample
SP5: feat, sample

45

40

35

35

35

25

20

Training Error on CIFAR-10

40

30

50
BA1: XY, batch (BN)
BA2: layer, batch
BA3: localXY, feat, batch
BA4: localXY, batch
BA5: feat, batch
BA6: batch
Our approach

45

40

Validation Error on CIFAR-10

Training Error on CIFAR-10

45

50
SP1: XY, sample
SP2: layer, sample (LN)
SP3: localXY, feat, sample
SP4: localXY, sample
SP5: feat, sample

30

25

20

40

30

25

20

35

30

25

20

15

15

15

15

10

10

10

10

5

0

10

20

5

30

0

10

Epoch

20

5

30

0

10

Epoch

20

BA1: XY, batch (BN)
BA2: layer, batch
BA3: localXY, feat, batch
BA4: localXY, batch
BA5: feat, batch
BA6: batch
Our approach

45

Validation Error on CIFAR-10

50

5

30

0

10

Epoch

20

30

Epoch

Figure 6: Different normalizations applied to a feedforward and convolutional network (shown in Figure 2 B).
All models were trained with 32 Samples per Batch (S/B), 1 Batch per Update (B/U). “Our approach” refers to
Streaming Normalization with “L2 norm” (Setting A with p=2 in Section 3.4) and α1 = β1 = 0.5, α2 = β2 = 0.5 and
β3 = 0 (see Section 3.3.3 for more details about hyperparameters). LN: Layer Normalization. Sample Normalizations
(including LN) seem to all work similarly. It seems beneficial to normalize each channel/feature map separately
(e.g., compare BA3 with BA4), like what BN does.

ResNet-like convolutional RNN: In Figure 7, we tested algorithms shown in Figure 1 C and D using the
architecture C in Figure 2. We also show the performance of our Streaming Normalization for reference.
50
SP1: XY, sample
SP2: layer, sample (LN)
SP3: localXY, feat, sample
SP4: localXY, sample
SP5: feat, sample

45

Validation Error on CIFAR-10

Training Error on CIFAR-10

40

35

30

25

20

50
BA1: XY, batch (BN)
BA2: layer, batch
BA3: localXY, feat, batch
BA4: localXY, batch
BA5: feat, batch
BA6: batch
Our Approach

45

40

40

35

35

Training Error on CIFAR-10

45

50
SP1: XY, sample
SP2: layer, sample (LN)
SP3: localXY, feat, sample
SP4: localXY, sample
SP5: feat, sample

30

25

20

40

30

25

20

35

30

25

20

15

15

15

15

10

10

10

10

5

0

10

20

Epoch

30

5

0

10

20

5

30

Epoch

0

10

20

Epoch

BA1: XY, batch (BN)
BA2: layer, batch
BA3: localXY, feat, batch
BA4: localXY, batch
BA5: feat, batch
BA6: batch
Our Approach

45

Validation Error on CIFAR-10

50

30

5

0

10

20

30

Epoch

Figure 7: Different normalizations applied to a recurrent and convolutional network (Figure 2 C with k1 = 5 and

k2 = 1). All models were trained with 32 Samples per Batch (S/B), 1 Batch per Update (B/U). “Our approach” refers
to Streaming Normalization with “L2 norm” (Setting A with p=2 in Section 3.4) and α1 = β1 = 0.5, α2 = β2 = 0.5
and β3 = 0 (see Section 3.3.3 for more details about hyperparameters). LN: Layer Normalization. Sample
Normalizations (including LN) seem to all work similarly. It seems beneficial to normalize each channel/feature
map separately (e.g., compare BA3 with BA4), like what BN does.

Densely Recurrent Convolutional Network: In Figure 8, we tested Time-Specific Batch Normalization and
14

Streaming Normalization on the architecture D in Figure 2.

30

30
TSBN
Ours

25

25

20

20

Validation Error on CIFAR-10

Training Error on CIFAR-10

TSBN
Ours

15

10

5

15

10

5

0

0
0

5

10

15

20

25

30

0

5

Epoch

10

15

20

25

30

Epoch

Figure 8: Time-specific Batch Normalization (TSBN) and Streaming Normalization applied to a densely recurrent

and convolutional network (Figure 2 D with k = 5). “Ours” refers to Streaming Normalization with “L2 norm”
(Setting B with p=2 in Section 3.4), 32 Samples per Batch (S/B) and 2 Batches per Update (B/U) and α1 = β1 = 0.7,
α2 = 0.3, β2 = 0 and β3 = 0.3 (see Section 3.3.3 for more details about hyperparameters). Sometimes for recurrent
networks, B/U > 1 is preferred, since the first mini-batch collects NormStats from all timesteps so that the second
mini-batch is normalized in a more stable way. TSBN was trained with 64 S/B, 1 B/U (32 S/B, 2 B/U would give
similar performance, if not worse). Streaming Normalization has similar performance to TSBN but does not require
storing different NormStats for each timestep.

7.5

More Experiments on Streaming Normalization

In Figure 9, we compare the performances of original BN (i.e., NormStats shared over time), time-specific BN, layer
normalization and streaming normalization on a recurrent and convolutional network shown in Figure 2 C.
We evaluated different choices of hyperparameter β1 ,β2 and β3 in Figure 10.

7.6

Recurrent Neural Networks for Character-level Language Modeling

We tried our simple implementations of vanilla RNN and GRU described in Section 5. The RNN and GRU both
have 1 hidden layer with 100 units. Weights are updated using the simple Manhattan update rule described in
[Liao et al., 2015]. The models were trained with learning rate 0.01 for 2 epochs and 0.001 for 1 epoch on a text
file of all Shakespeare’s work concatenated. We use 99% the text file for training and 1% for validation. The
training and validation softmax losses are reported. Training losses are from mini-batches so they are noisy, and we
smoothed them using moving averages of 50 neighbors (using the Matlab smooth function). The test loss on the
entire validation set is evaluated and recorded every 20 mini-batches. We show in Figure 11 and 12 the performances
of Time-specific Batch Normalization, Layer Normalization and Streaming Normalization. Truncated BPTT was
performed with 100 timesteps.

15

100

90

90

80

80
Original BN
Time-specific BN
LN
Ours

50

40

60

50

40

30

30

20

20

10

0

Original BN
Time-specific BN
LN
Ours

Training Error on CIFAR-10

60

30
Original BN
Time-specific BN
LN
Ours

70

Validation Error on CIFAR-10

Training Error on CIFAR-10

70

30

Original BN
Time-specific BN
LN
Ours

25

25

20

20

Validation Error on CIFAR-10

100

15

15

10

10

5

5

10

0

10

20

Epoch

30

0

0

10

20

0

30

Epoch

0

10

20

Epoch

30

0

0

10

20

30

Epoch

Figure 9: Different normalizations applied to a recurrent and convolutional network (shown in Figure 2 C with

unrolling parameters k1 = k2 = 5). The right two pannels are zoomed-in versions of the left two pannels. “Ours”
refers to Streaming Normalization with “L2 norm” (Setting B with p=2 in Section 3.4), 32 Samples per Batch (S/B),
2 Batches per Update (B/U) and α1 = β1 = 0.7, α2 = 0.3, β2 = 0 and β3 = 0.3 (see Section 3.3.3 for more details
about hyperparameters). Time-specific Batch Normalization, original BN and Layer Normalization (LN) were
trained with 64 S/B, 1 B/U (32 S/B, 2 B/U would give similar performance, if not worse). Streaming Normalization
clearly outperforms other methods in training. Streaming Normalization converges more than twice as fast as
LN. Note that 32 S/B 2 B/U and 64 S/B 1 B/U are equivalent to LN since it processes samples independently.
Original BN fails on testing.

16

80

70

70

60

50

40

60

50

40

30

30

20

20

10

No Grad (0,0,0)
Mini-batch Grad (0,0,1)
Current Update Grad (0,1,0)
Previous Update Grad (1,0,0)
One combination (0.7,0.3,0)
Another combination (0.7,0,0.3)

Training Error on CIFAR-10

80

30

No Grad (0,0,0)
Mini-batch Grad (0,0,1)
Current Update Grad (0,1,0)
Previous Update Grad (1,0,0)
One combination (0.7,0.3,0)
Another combination (0.7,0,0.3)

90

Validation Error on CIFAR-10

Training Error on CIFAR-10

90

0

30

100
No Grad (0,0,0)
Mini-batch Grad (0,0,1)
Current Update Grad (0,1,0)
Previous Update Grad (1,0,0)
One combination (0.7,0.3,0)
Another combination (0.7,0,0.3)

No Grad (0,0,0)
Mini-batch Grad (0,0,1)
Current Update Grad (0,1,0)
Previous Update Grad (1,0,0)
One combination (0.7,0.3,0)
Another combination (0.7,0,0.3)

25

25

20

20

Validation Error on CIFAR-10

100

15

15

10

10

5

5

10

0

10

20

Epoch

30

0

0

10

20

0

30

Epoch

0

10

20

Epoch

30

0

0

10

20

30

Epoch

Figure 10: Evaluate different choices of hyperparameter β1 ,β2 and β3 . The architecture is a recurrent and

convolutional network (shown in Figure 2 C with unrolling parameters k1 = k2 = 5). The right two pannels are
zoomed-in versions of the left two pannels. The models are Streaming Normalization with “L2 norm” (Setting B
with p=2 in Section 3.4), 32 Samples per Batch (S/B), 2 Batches per Update (B/U) and α1 , α2 = 0.3, κ1 = κ3 =
0.7, κ2 = κ4 = 0.3. The hyperparameters (β1 , β2 , β3 ) are shown in the figure. (0,0,0) means that the gradients of
NormStats are ignored. (0,0,1) means only using NormStats gradients from the current mini-batch. (0,1,0) means
only using NormStats gradients accumulated since the last weight update. Note that regardless the values of β,
the gradients of NormStats are always accumulated (See Section 3.3.2). Using gradients from the previous weight
update (i.e., 1,0,0) seems to work reasonably well. Some combinations (i.e., (0.7,0.0.3) or (0.7,0.0.3)) of previous
and current gradients seem to give the best performances. This experiment indicates that streaming the gradients
of NormStats is very important for performance.

17

3.4

4.5
TSBN-RNN
LN-RNN
Ours

TSBN-RNN
LN-RNN
Ours

3.2
4

3

Validation Loss

Training Loss

3.5

3

2.8

2.6

2.4
2.5

2.2

2
2

1.5
10 -4

10 -3

10 -2

10 -1

10 0

1.8
10 -2

10 1

10 -1

Epoch

10 0

10 1

Epoch

Figure 11: Character-level language modeling with RNN on Shakespeare’s work concatenated. The training (left)

and validation (right) softmax losses are reported. “Ours” refers to Streaming Normalization with “L2 norm”
(Setting B with p=2 in Section 3.4), 32 Samples per Batch (S/B), 2 Batches per Update (B/U) and α1 = β1 = 0.7,
α2 = 0.3, β2 = 0 and β3 = 0.3 (see Section 3.3.3 for more details about hyperparameters). TSBN: time-specific BN.
LN: Layer Normalization. Both TSBN and Streaming Normalization (SN) converges faster than LN. SN reaches
slightly lower loss than TSBN and LN.

4.5

3.4
TSBN-GRU
LN-GRU
Ours

TSBN-GRU
LN-GRU
Ours

3.2

4
3

2.8

Validation Loss

Training Loss

3.5

3

2.6

2.4

2.2

2.5
2

1.8
2

1.6

1.5
10 -4

10 -3

10 -2

10 -1

10 0

1.4
10 -2

10 1

Epoch

10 -1

10 0

10 1

Epoch

Figure 12: Character-level language modeling with GRU on Shakespeare’s work concatenated. The training (left)

and validation (right) softmax losses are reported. “Ours” refers to Streaming Normalization with “L2 norm”
(Setting B with p=2 in Section 3.4), 32 Samples per Batch (S/B), 2 Batches per Update (B/U) and α1 = β1 = 0.7,
α2 = 0.3, β2 = 0 and β3 = 0.3 (see Section 3.3.3 for more details about hyperparameters). TSBN: time-specific BN.
LN: Layer Normalization. Streaming Normalization converges faster than LN and reaches lower loss than TSBN.

18

8

Discussion

Biological Plausibility
We found that the simple “Neuron-wise normalization” (BA6 in Figure 1 D) performs very well (Figure 6 and 7). This
setting does not require collecting normalization statistics from any other neurons. We show the streaming version
of neuron-wise normalization in Figure A1, and the performance is again competitive. In neuron-wise normalization,
each neuron simply maintains running estimates of its own mean and variance (and related gradients), and all the
information is maintained locally. This approach may serve as a baseline model for biological homeostatic plasticity
mechanisms (e.g., Synaptic Scaling) [Turrigiano and Nelson, 2004, Stellwagen and Malenka, 2006, Turrigiano, 2008],
where each neuron internally maintains some normalization/scaling factors that depend on neuron’s firing history
and can be applied and updated in a pure online fashion.
Lp Normalization
Our observations about Lp normalization have several biological implications: First, we show that most Lp
normalizations work similarly, which suggests that there might exist a large class of statistics that can be used
for normalization. Biological systems could implement any of these methods to get the same level of performance.
Second, L1 normalization is particularly interesting, since its gradient computations are much easier for biological
neurons to implement.
As an orthogonal direction of research, it would also be interesting to study the relations between our Lp normalization
(standardizing the average Lp norm of activations) and Lp regularization (discounting the the Lp norm of weights,
e.g., L1 weight decay).
Theoretical Understanding
Although normalization methods have been empirically shown to significantly improve the performance of deep
learning models, there is not enough theoretical understanding about them. Activation-normalized neurons behave
more similarly to biological neurons whose activations are constrained into a certain range: is it a blessing or a curse?
Does it affect approximation bounds of shallow and deep networks [Mhaskar et al., 2016, Mhaskar and Poggio, 2016]?
It would also be interesting to see if certain normalization methods can mitigate the problems of poor local minima
and saddle points, as the problems have been analysed without normalization [Kawaguchi, 2016].
Internal Covariant Shift in Recurrent Networks
Note that our approach (and perhaps the brain’s “synaptic scaling”) does not normalize differently for each timestep.
Thus, it does not naturally handle internal covariant shift [Ioffe and Szegedy, 2015] (more precisely, covariate shift
over time) in recurrent networks, which was the main motivation of the original Batch Normalization and Layer
Normalization. Our results seem to suggest that internal covariate shift is not as hazardous as previously believed
as long as the entire network’s activations are normalized to a good range. But more research is needed to answer
this question.

Acknowledgments
This work was supported by the Center for Brains, Minds and Machines (CBMM), funded by NSF STC award CCF
– 1231216.

References
[Amodei et al., 2015] Amodei, D., Anubhai, R., Battenberg, E., Case, C., Casper, J., Catanzaro, B., Chen, J.,
Chrzanowski, M., Coates, A., Diamos, G., et al. (2015). Deep speech 2: End-to-end speech recognition in english
and mandarin. arXiv preprint arXiv:1512.02595.
19

[Ba et al., 2016] Ba, J. L., Kiros, J. R., and Hinton, G. E. (2016).
arXiv:1607.06450.

Layer normalization.

arXiv preprint

[Chung et al., 2014] Chung, J., Gulcehre, C., Cho, K., and Bengio, Y. (2014). Empirical evaluation of gated
recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555.
[Cooijmans et al., 2016] Cooijmans, T., Ballas, N., Laurent, C., and Courville, A. (2016). Recurrent batch normalization. arXiv preprint arXiv:1603.09025.
[He et al., 2015] He, K., Zhang, X., Ren, S., and Sun, J. (2015). Deep residual learning for image recognition. arXiv
preprint arXiv:1512.03385.
[Ioffe and Szegedy, 2015] Ioffe, S. and Szegedy, C. (2015). Batch normalization: Accelerating deep network training
by reducing internal covariate shift. arXiv preprint arXiv:1502.03167.
[Kawaguchi, 2016] Kawaguchi, K. (2016). Deep learning without poor local minima. In Advances in Neural
Information Processing Systems (NIPS). to appear.
[Laurent et al., 2015] Laurent, C., Pereyra, G., Brakel, P., Zhang, Y., and Bengio, Y. (2015). Batch normalized
recurrent neural networks. arXiv preprint arXiv:1510.01378.
[Liao et al., 2015] Liao, Q., Leibo, J. Z., and Poggio, T. (2015). How important is weight symmetry in backpropagation? arXiv preprint arXiv:1510.05067.
[Liao and Poggio, 2016] Liao, Q. and Poggio, T. (2016). Bridging the gaps between residual learning, recurrent
neural networks and visual cortex. arXiv preprint arXiv:1604.03640.
[Mhaskar et al., 2016] Mhaskar, H., Liao, Q., and Poggio, T. (2016). Learning real and boolean functions: When is
deep better than shallow. arXiv preprint arXiv:1603.00988.
[Mhaskar and Poggio, 2016] Mhaskar, H. and Poggio, T. (2016). Deep vs. shallow networks: An approximation
theory perspective. arXiv preprint arXiv:1608.03287.
[Neyshabur et al., 2015] Neyshabur, B., Salakhutdinov, R. R., and Srebro, N. (2015). Path-sgd: Path-normalized
optimization in deep neural networks. In Advances in Neural Information Processing Systems, pages 2422–2430.
[Salimans and Kingma, 2016] Salimans, T. and Kingma, D. P. (2016). Weight normalization: A simple reparameterization to accelerate training of deep neural networks. arXiv preprint arXiv:1602.07868.
[Stellwagen and Malenka, 2006] Stellwagen, D. and Malenka, R. C. (2006). Synaptic scaling mediated by glial tnf-α.
Nature, 440(7087):1054–1059.
[Turrigiano and Nelson, 2004] Turrigiano, G. and Nelson, S. (2004). Homeostatic plasticity in the developing
nervous system. Nature Reviews Neuroscience, 5(2):97–107.
[Turrigiano, 2008] Turrigiano, G. G. (2008). The self-tuning neuron: synaptic scaling of excitatory synapses. Cell,
135(3):422–435.
[Ullman and Schechtman, 1982] Ullman, S. and Schechtman, G. (1982). Adaptation and gain normalization.
Proceedings of the Royal Society of London B: Biological Sciences, 216(1204):299–313.
[Vedaldi and Lenc, 2015] Vedaldi, A. and Lenc, K. (2015). Matconvnet: Convolutional neural networks for matlab.
In Proceedings of the 23rd Annual ACM Conference on Multimedia Conference, pages 689–692. ACM.

20

A

Other Variants of Streaming Normalization

40

40
BA6-B
BA6-S
BA4-B
BA4-S
BA1-B
BA1-B

35

35

30

Validation Error on CIFAR-10

Training Error on CIFAR-10

30

25

20

25

20

15

15

10

10

5

BA6-B
BA6-S
BA4-B
BA4-S
BA1-B
BA1-B

0

5

10

15

20

25

5

30

Epoch

0

5

10

15

20

25

30

Epoch

Figure A1: We explore other variants of Streaming Normalization with different NormRef (e.g., SP1-SP5, BA1-BA6

in 1 C and D) within each mini-batch. -B denotes the batch version. -S denotes the streaming version. The
architecture is a feedforward and convolutional network (shown in Figure 2 B). Streaming significantly lowers
training errors.

21

