arXiv:1804.00815v1 [cs.LG] 3 Apr 2018

Convolutional Neural Networks Regularized by
Correlated Noise
Shamak Dutta, Bryan Tripp

Graham W. Taylor

Systems Design Engineering & Center for Theoretical Neuroscience
University of Waterloo
Waterloo, Canada
{s7dutta,bptripp}@uwaterloo.ca

School of Engineering, University of Guelph
Vector Institute for Artificial Intelligence
Canadian Institute for Advanced Research
gwtaylor@uoguelph.ca

Abstract—Neurons in the visual cortex are correlated in
their variability. The presence of correlation impacts cortical
processing because noise cannot be averaged out over many
neurons. In an effort to understand the functional purpose of
correlated variability, we implement and evaluate correlated
noise models in deep convolutional neural networks. Inspired
by the cortex, correlation is defined as a function of the distance
between neurons and their selectivity. We show how to sample
from high-dimensional correlated distributions while keeping the
procedure differentiable, so that back-propagation can proceed
as usual. The impact of correlated variability is evaluated on
the classification of occluded and non-occluded images with and
without the presence of other regularization techniques, such
as dropout. More work is needed to understand the effects
of correlations in various conditions, however in 10/12 of the
cases we studied, the best performance on occluded images was
obtained from a model with correlated noise.
Index Terms—Correlated Variability, Convolutional Neural
Networks, Regularization, Stochastic Neurons

I. I NTRODUCTION
Convolutional neural networks (CNN) trained for object
recognition tasks are similar to the visual cortex in many ways.
For example, early layers show Gabor-like receptive fields
similar to V1 [1], late layers that are highly predictive of V4,
and inferior temporal cortex (IT) responses [2]. However, these
networks lack the correlated variability of neuron responses
in the human brain, among other major differences. In this
paper, we discuss methods to incorporate correlated variability
into deep convolutional networks and analyze its effect on
recognition performance.
Studying stochastic neurons is interesting because the effect
of stochasticity on learning and computation in artificial neural
systems may help us in modeling biological neurons. In
population coding schemes in the brain, the joint activities of
many neurons encode the value of a quantity. One advantage
of population coding is that the noise can be averaged out
over many neurons. However, the observation of correlation
in spike variability [3] is concerning because it can prevent
noise averaging and strongly affect cortical processing. The
function of correlated variability in the brain is unclear. Our
goal is to understand if correlated variability has benefits
that can be realized in artificial neural networks by studying
its influence on certain tasks. One example of such a task
is the long-standing problem of object recognition under

partial occlusion. This requires the ability to robustly model
invariance to certain transformations of the input. The primary
method of gaining invariance to transformations is data driven,
either by attempting to collect instances of the object under a
wide variety of conditions, or augmenting the existing dataset
via synthetic transformations. The authors in [4] suggest that
the right way to learn invariance is not by adding data, as
occlusions follow a long-tail distribution which cannot be
covered, even by large-scale efforts in collecting data. This
motivates the need to modify the structure of the network
to learn invariance. We hypothesize that correlated variability
might improve recognition performance in the challenging
setting of partial occlusion.
A common approach for regularizing deep neural networks
is to inject noise during training; for example, adding or
multiplying noise to the hidden units of the neural network,
like in dropout [5]. Most solutions include additive or multiplicative independent noise in the hidden units. This is
widely used because of its simplicity and effectiveness. We
are motivated to consider correlated noise that is dependent
on the weights of the network, such as our proposal to
add noise sampled from a correlated distribution, where the
correlation is a function of the differences in spatial position
and selectivity of neurons, similar to the visual cortex. One
of the major concerns of stochasticity in neural networks is
the tendency to break differentiability, which prevents gradient computation via back-propagation. Depending on the
noise distribution, one may simply ignore stochasticity in
back-propagation (e.g. the straight-through estimator in the
case of binary stochastic neurons [6]), or one may apply
computationally convenient estimators for non-differentiable
modules. For the latter case, there has been much work focused
on back-propagation through non-differentiable functions. For
example, the re-parameterization trick in [7] allows for backpropagation through sampling from a broad class of continuous distributions. Re-parameterization using the Gumbelmax trick and the softmax function in [8], [9] allows for
back-propagation through samples from categorical discrete
distributions. Advancements in the area of back-propagation
through non-differentiable modules is relevant to our work
because it allows us to consider many interesting types of
noise models.

In this paper, we discuss four different types of noise models: independent Gaussian, correlated Gaussian, independent
Poisson, and correlated Poisson (Section III-A). In the case
of correlation, we also describe how to construct correlation
between samples (Section III-B). Finally, we evaluate these
different noise models on the classification of occluded and
non-occluded images.
II. R ELATED W ORK
Independent Noise Models: The analysis of noise in deep
networks has focused on models that use independent noise
to perturb the activations of neurons. For example, dropout
[5] is an effective method to regularize neural networks where
each unit is independently dropped with a certain probability.
Independent Gaussian noise has also been extensively explored
in [10], where noise is added to the input, or before or
after the non-linearity in other layers. The authors connect
different forms of independent noise injection to certain forms
of optimization penalties in a special form of a denoising
autoencoder. An observation to note is that additive Gaussian
noise with zero-mean and variance equal to the unit activation
relates to a penalty that encourages sparsity on the hidden
unit activations. This is interesting because the time intervals
between spikes in a cortical neuron are irregular and can be
modeled using a Poisson process, such that the variance of the
spike count in a small time interval is equal to the mean spike
count. This motivates us to investigate whether we can model
artificial neurons as samples from a Poisson distribution.
The work done in [10] focused on unsupervised learning in
autoencoders and used the learned hidden representations as
inputs to a classifier. Our approach differs in the fact that we
are directly using a supervised classifier (CNN) to analyze the
injection of noise.
Back-propagation Through Random Nodes: Gradientbased learning which leverages back-propagation in neural
networks requires that all operations that depend on the trainable parameters be differentiable. Stochastic neural networks
usually involve samples from a distribution on the forward
pass. However, the difficulty is that we cannot back-propagate
through the sampling operation. It is shown in [6] that in
the case of injection of additive or multiplicative noise in
a computational graph, gradients can be computed as usual.
We use this method in Section III-A1. Bengio et al. [6] also
introduce the concept of a straight-through estimator, where a
copy of the gradient with respect to the stochastic output is
used directly as an estimator of the gradient with respect to the
sigmoid (or any non-linearity) operator. Similar to the straightthrough estimator, in Sections III-A3 and III-A4, we use the
expected value of the Poisson distribution as if it were the
output of the neuron during back-propagation. The work in [8],
[9] allows for back-propagation through samples from discrete
categorical distributions. While we do not use this technique in
our work, it is possible to choose an upper bound K to convert
a Poisson distribution to a categorical distribution of size K.
The sample from the distribution would be the expected value

of this estimated categorical distribution and back-propagation
can proceed since the entire process is made differentiable.
Augmentation in Feature Space: Dataset augmentation is
a cheap and effective way to generate more training data with
variability that is expected at inference time. Recently, some
works have considered augmentation techniques, such as the
addition of noise, as well as interpolation and extrapolation
from pairs of examples, not in input space, but in a learned
feature space [11], [12]. However, [11] shows that simple
noise models (e.g. Gaussian) do not work effectively when
compared to extrapolation. We are motivated by the fact that
more sophisticated noise models (e.g. correlated) could be
useful for feature space-based augmentation.
III. M ETHODS
A. Noise Framework
We analyzed different types of independent and correlated
noise to elucidate their effects on neural networks. This helped
us understand the impact of injecting correlated noise when
compared to the common practice of injecting independent
noise.
1) Independent Gaussian Noise: We consider hi to be the
output of a stochastic neuron. The output is a deterministic
function of a differentiable transformation ai and a noise
source zi , as considered in [6]. ai is typically a transformation
of its inputs (vector output of other neurons) and trainable
parameters (weights and biases of a network). The output of
the stochastic neuron is:
hi = f (ai , zi ).

(1)

As long as hi is differentiable with respect to ai and has
a non-zero gradient, we can train the network with backpropagation. In this section, one form of noise we consider
is additive independent Gaussian noise with zero mean and
σ 2 variance, where:
f (ai , zi ) = ai + zi ,

(2)

zi ∼ N (0, σ 2 ).

(3)

At test time, we can compute the expectation of the noisy
activations by sampling from the distribution N (ai , σ 2 ); however, this can be computationally expensive for large datasets.
Instead, we can approximate the expectation by scaling the
units by their expectation, as in dropout [5]. Since we are
using zero mean additive noise, no scaling is required, as the
expectation does not change.
A special case of Equation 3 is when σ 2 = ai . The
distribution of activations for a specific stimulus will follow
N (ai , ai ), which has a Fano factor of 1. This is similar to
biological neurons, which exhibit Poisson-like statistics with
a Fano factor of approximately 1. It also means that zi is
now a function of ai and the gradient of zi with respect to ai
exists through re-parameterization. A sample from a normal
distribution N (µ, σ 2 ) can be constructed as:

(4)

√
In the case when σ 2 = ai and µ = 0, zi = ai xi . In
practice, the gradients can be unstable for the square-root
function for small values. To solve this problem, we add a
small value of ǫ = 0.0001 to the argument of the square-root
function.
2) Correlated Gaussian Noise: We consider h ∈ IRn to be
a vector of outputs of n stochastic neurons. h is a sum of
a noise vector z ∈ IRn and a vector output a ∈ IRn , which
is a differentiable transformation of its inputs and trainable
parameters, similar to Section III-A1. The vector output is:
h = z + a,

(5)

z ∼ N (0, Σ).

(6)

Given a desired mean µ and correlation matrix Σ, z ∈ IRn
is sampled as follows:
– Sample X ∼ N (0, In )
– Compute the Cholesky decomposition: Σ = LL∗ , where
L∗ is the conjugate transpose of L.
– z = µ + LX
– If σ ∈ IRn is the desired standard deviation, then
z = diag(σ)(µ + LX),

(7)

where diag(σ) ∈ IRn×n is a matrix with the standard
deviations on the diagonal and zeros elsewhere.
As described in Section III-B2, the correlation matrix Σ is a
function of the trainable weights of the network. This implies
that the entire sampling process is differentiable with respect
to the parameters µ, σ, and the trainable weights, and that
gradient-based learning can proceed as usual.
3) Independent Poisson Noise: We consider hi to be the
output of a stochastic neuron. The definition of ai is the same
as in Section III-A1. In the case of independent Poisson noise,
the output of the neuron is:
hi ∼ Poisson(λ = ai ),

(8)

where the mean is given by ai . This poses a problem in
back-propagation, as there is no gradient of hi with respect
to the parameter λ = ai . The re-parameterization trick in [7]
cannot be applied in this case because the Poisson distribution
is discrete. To avoid this problem, we set the output of the
unit to its mean value during the backward pass, which is
ai . We still propagate the sample from the distribution on the
forward pass. This is similar to the straight-through estimator
for back-propagation through stochastic binary neurons [6].
The Gumbel-softmax method [8], [9] can be used here if the
Poisson distribution is converted to a categorical distribution
using an upper threshold K. We plan to explore this avenue
in future work.

h ∼ Poisson(λ, Σ).

(9)

We draw approximate samples from this distribution in the
following way. Given a desired mean λ ∈ IRn and correlation
matrix Σ ∈ IRn×n :
– Sample X ∼ N (µ, Σ), where µ is chosen arbitrarily
– Apply the univariate normal cumulative distribution
function (CDF): Y = Fx (X; µ = 0, σ = 1)
– Apply the quantile (inverse CDF) of the Poisson distribution: z ∼ Fz−1 (Y ; λ)
This sampling procedure is non-differentiable for a correlated Poisson. As in Section III-A3, we estimate the gradient
using the mean λ on the backward pass. The actual correlations between Poisson variables depend on the rates λ and are
smaller than the desired correlation matrix Σ.

1.00
0.30

0.75

0.27

0.50
kernel similarity

xi ∼ N (0, 1),
zi = µ + σxi .

4) Correlated Poisson Noise: Similar to Section III-A2,
h ∈ IRn is a vector of outputs of n stochastic neurons. h
is a sample from a correlated Poisson distribution of mean λ
and correlation matrix Σ,

0.24

0.25

0.21

0.00

0.18

−0.25
−0.50

0.15

−0.75

0.12

−1.00

0

10
20
30
euclidean coordinate distance

40

0.09

Fig. 1. Correlation dependence on distance and tuning similarity for a =
0.225, b = 0.0043, c = 0.09 and τ = 1.87. The correlation value of the
color plot is indicated by the bar on the right.

B. Correlation Matrix
In this section, we discuss how the correlation matrix Σ is
constructed, which is used in Sections III-A2 and III-A4. Note
that the neuron equivalent in a CNN is a unit in the output
feature map of a given layer.
Neurons in the cortex are correlated in their stochasticity.
This correlation is a function of the spatial spread and tuning
similarity [13]. In the visual cortex, correlations have been
studied between neurons in the same visual area. Analogously,
we consider correlations between units in the same layer of a
CNN. The details of spatial similarity and tuning similarity are
described in Sections III-B1 and III-B2. For a given layer in a
convolutional network, if the width and height of the feature
maps, as well as the number of feature maps are defined
as w, h, and k, respectively, then the dimension d of the

correlated distribution we draw samples from is d = whk
and the correlation matrix is Σ ∈ IRwhk×whk .
Similar to a relationship suggested in [13], the correlation
between two neurons, x1 and x2 , is determined as:
f (x1 , x2 ) = [a − b(d(x1 , x2 ))]+ · e

k(x1,x2)−1
τ

+ c,

(10)

where [·]+ is max(·, 0), d(·, ·) is a function that returns the
scaled Euclidean distance between two neurons, k(·, ·) is a
function that returns the tuning similarity between two neurons
(bounded in [−1, 1]), and a, b, c, and τ are hyper- parameters.
The correlation is summarized in Figure 1 for specific values
of a, b, c, and τ . To summarize, within a layer of a CNN,
every neuron is correlated to every other neuron in the same
layer by a value determined by how far apart they are and
their tuning similarity.
1) Spatial Similarity: The spatial similarity between two
units in any output feature map within a layer is determined
by the Euclidean distance between the coordinates of the
neurons within their respective feature maps. The spatial
distance between two neurons that are a part of the ith and
j th feature maps, respectively, with coordinates p = (xi , yi )
and q = (xj , yj ) is:
d(p, q) =

q
(xi − xj )2 + (yi − yj )2 ,

(11)

Since the dimensions of the feature map do not change as
training progresses, we can pre-compute the spatial correlations for all pairs of neurons before training begins.
2) Tuning Similarity: The tuning similarity between two
units in any output feature maps within a layer is determined
by the cosine similarity of the normalized weight matrices
that produced them. Consider a weight tensor (kernel in a
convolutional network) of dimension d = k × k × m × n,
where k is the kernel size, m is the number of input channels
from the previous layer, and n is the number of kernels. The
ith kernel wi is of dimension d = kkm × 1. The tuning
similarity between two neurons in the ith and j th feature maps
is determined as:

TABLE I
M ODIFIED VERSION OF A LL C ONVNET (ALL-CNN-C) ARCHITECTURE
WITH 10 FILTERS IN LAYER 1 USED WITH CIFAR-10.
Input: 32 × 32 RBG image
Layer 1: 3 × 3 conv. 10 filters, ReLU
Layer 2: 3 × 3 conv. 96 filters, ReLU
Layer 3: 3 × 3 conv. 96 filters, stride = 2, ReLU
Layer 4: 3 × 3 conv. 192 filters, ReLU
Layer 5: 3 × 3 conv. 192 filters, ReLU
Layer 6: 3 × 3 conv. 192 filters, stride = 2, ReLU
Layer 7: 3 × 3 conv. 192 filters, ReLU
Layer 8: 1 × 1 conv. 192 filters, ReLU
Layer 9: 1 × 1 conv. 10 filters, ReLU
Layer 10: global averaging over 6 × 6 spatial dimensions
10-way softmax

TABLE II
S TOCHASTIC MODELS EVALUATED AS PART OF LAYER 1
Model name
AllConvNet baseline
AllConvNet ind. Gaussian σ = 1.0
AllConvNet ind. Gaussian σ = ai
AllConvNet correlated Gaussian
AllConvNet ind. Poisson
AllConvNet correlated Poisson

TESTS .

abbreviation
Baseline
IG A
IG B
CG
IP
CP

Original

Checkerboard

Central Occlusion

Horizontal Half

Horizontal Bars

Vertical Bars

Vertical Half

Fig. 2. Different types of occlusions used to evaluate recognition performance.
Images shown are taken from CIFAR-10.

k(xi , xj ) =



wi
kwi k

T

·

wj
,
kwj k

(12)

since we are calculating the cosine similarity, k(·, ·) ∈ [−1, 1].
Note that the tuning similarity solely depends on the feature
maps that the output units are a part of.
IV. E XPERIMENTS
We performed preliminary experiments with the aforementioned noise models using an architecture equivalent to the
AllConvnet network [14] (All-CNN-C) with one exception:
the first layer contains 10 feature maps instead of 96, as shown
in Table I. This was done for computational tractability, as the
correlation matrix grows as (whk)2 , where w and h are the
width and height of a feature map, and k is the number of
feature maps in a layer. As a result, the sampling procedure
from a correlated distribution can slow training by a large
amount. While we do not aim for state-of-the-art results, our
goal is to obtain a model that achieves respectable performance
on CIFAR-10 and analyze the effect of adding various kinds
of noise. All models are trained for 100 epochs and the final
chosen model in each case is the one with the lowest validation
loss during training to prevent overfitting.
The CIFAR-10 dataset is used for all experiments, split into
50,000 training images and 10,000 test images. The training
set does not contain any occluded images. All experiments

TABLE III
R ECOGNITION PERFORMANCE OF LAYER 1
Model

Baseline
IG A
IG B
CG
IP
CP

Test Set
mean
(%)
75.5
72.6
75.9
80.8
71.8
76.8

s.d.
(%)
0.2
0.3
0.1
0.1
0.8
0.5

Central Occlusion
mean
(%)
62.1
61.6
63.3
69.1
59.6
64.3

s.d.
(%)
0.1
0.2
0.1
0.1
0.5
0.2

STOCHASTIC BEHAVIOURAL MODELS WITH NO ADDITIONAL REGULARIZATION OVER

Checker Board

Horizontal Bars

mean
(%)
46.8
45.5
49.6
58.1
44.6
47.5

mean
(%)
35.7
36.1
41.3
51.5
35.2
44.7

s.d.
(%)
0.3
0.2
0.2
0.2
0.6
0.3

s.d.
(%)
0.3
0.3
0.4
0.4
0.7
0.3

Vertical Bars
mean
(%)
34.8
33.0
35.1
40.6
32.7
37.6

s.d.
(%)
0.4
0.2
0.4
0.1
0.4
0.2

Horizontal Half
mean
(%)
41.1
43.0
40.6
47.4
40.2
44.6

s.d.
(%)
0.4
0.3
0.6
0.3
0.4
0.4

10 RUNS
Vertical Half

mean
(%)
40.3
39.5
37.2
41.6
36.3
45.5

s.d.
(%)
0.3
0.3
0.3
0.3
0.4
0.2

TABLE IV
R ECOGNITION PERFORMANCE OF LAYER 1 STOCHASTIC BEHAVIOURAL MODELS WITH DROPOUT OVER 10 RUNS .
Model

Baseline
IG A
IG B
CG
IP
CP

Test Set
mean
(%)
84.2
85.1
84.5
84.2
84.4
84.9

s.d.
(%)
0.1
0.1
0.1
0.1
0.2
0.2

Central Occlusion
mean
(%)
73.8
74.7
74.6
75.5
75.1
75.4

s.d.
(%)
0.1
0.1
0.1
0.1
0.1
0.1

Checker Board

Horizontal Bars

mean
(%)
63.1
67.3
66.9
58.4
63.8
68.1

mean
(%)
46.9
50.2
50.1
54.1
52.3
53.1

s.d.
(%)
0.1
0.1
0.1
0.2
0.1
0.3

were performed using the TensorFlow framework [15]. We
evaluate classification performance on the test set, including
occluded versions of the test set, as shown in Figure 2.
We first experiment by incremental addition of the stochastic
behaviour to understand its effect. The baseline model is
described in Table I. First, we add noise in layer 1 of the
AllConvNet architecture. The different noise architectures for
layer 1 are shown in Table II. The results of this experiment
are shown in Section IV-A.
We then analyze whether the benefits of noise can be
realized in the presence of another regularizer by evaluating
the effect of noise in the presence of dropout. In this case,
the baseline model shown in Table I is trained with dropout
in specific layers. As before, we add different types of noise
to layer 1 in order to understand its impact, as described in
Section IV-B.
A. Absence of Dropout
The classification performance of the models that do not
incorporate dropout are summarized in Table III. The models
are abbreviated, as shown in Table II.
Independent Gaussian noise does not seem to have an
appreciable effect on the learning of the network. We varied σ
in the set {0.1, 0.5, 1.0, 1.5, 2.0} to find the network with the
optimal variance, which was σ = 1.0. It is possible that only
adding independent Gaussian noise to one layer is not enough
to regularize the network. This motivates the need to add it to
multiple layers in order to understand its true effect.
Correlated Gaussian noise achieves the best results at classification of the clean test set, and on five out of the six occlusion
classes. The improvement from the baseline model is ∼ 5%
after being added only in the first layer, which is encouraging.
To ensure that the classification improvement was consistent

s.d.
(%)
0.2
0.3
0.3
0.2
0.2
0.3

Vertical Bars
mean
(%)
55.1
55.4
56.4
57.5
58.2
57.0

s.d.
(%)
0.2
0.2
0.2
0.2
0.1
0.1

Horizontal Half
mean
(%)
47.5
50.6
49.7
47.4
49.9
47.9

s.d.
(%)
0.2
0.2
0.2
0.2
0.2
0.2

Vertical Half
mean
( %)
43.3
46.1
45.8
47.2
43.2
44.6

s.d.
(%)
0.3
0.3
0.2
0.2
0.2
0.3

across all classes and not only for some outliers, we ran the
breakdown of the class predictions for each set shown in
Table V. It can be seen that the improvements are consistent
for ∼ 70% of different output classes.
Independent Poisson noise has worse performance than
the baseline model across all sets. To investigate why this
occurred, we examined the distribution of the activations of
layer 1 (samples from the Poisson distribution). The values
were highly variable in the range [3, 35]. This may stem from
the fact that we back-propagate the mean rate of the Poisson
instead of the actual sample value, which could affect learning.
We hypothesize that a penalty on the activations, combined
with the independent Poisson noise could achieve acceptable
classification results.
The correlated Poisson model also achieves better results
than the baseline model across the test set and all occlusion
classes and has the best result on the vertical half occlusion set.
It also performs better compared to the independent Poisson
model across all occlusion sets showing that correlations have
an appreciable effect on the classification performance.
B. Presence of Dropout
Dropout was applied to the input image, as well as after each
of the layers that has a stride of 2 (simply a convolutional
layer that replaces pooling [14], specifically layer 3 and 6).
The dropout probability at the input was 20% and was 50%
otherwise.
The classification performance of the models that incorporate dropout are summarized in Table IV, with the model
abbreviations shown in Table II.
We observe that, when compared to the model without
dropout, the baseline model with dropout performed better

TABLE V
C LASS P REDICTION A CCURACY FOR (A) A LL C ONV N ET BASELINE MODEL WITHOUT DROPOUT AND (B) A LL C ONV N ET C ORRELATED G AUSSIAN
MODEL WITHOUT DROPOUT.
Class
airplane
automobile
bird
cat
deer
dog
frog
horse
ship
truck

Test Set
A

B

78.9
87.7
69.4
59.6
79.6
70.5
85.9
79.2
88.6
87.1

78.2
89.0
73.2
62.5
82.2
76.1
88.2
83.5
87.8
89.7

Central Occlusion
A
B
73.3
79.6
57.7
35.2
68.8
60.4
74.5
71.1
71.9
70.5

66.6
83.1
58.7
48.7
63.4
72.6
75.7
75.1
73.1
76.4

Checker Board
A
B

Horizontal Bars
A
B

54.0
18.1
46.2
37.8
63.9
48.4
64.0
40.3
70.0
67.9

76.9
42.1
32.1
52.2
63.1
32.5
15.0
16.9
51.2
33.3

52.8
29.7
51.9
44.2
69.7
58.0
71.4
51.4
70.4
60.7

56.1
39.0
38.2
44.6
73.6
66.7
42.1
36.9
73.6
26.2

Vertical Bars
A
B
51.5
18.4
52.8
54.8
50.4
48.3
16.9
37.8
53.8
31.2

59.3
18.5
57.1
45.7
50.7
68.3
23.1
44.3
47.1
35.2

Horizontal Half
A
B
70.1
62.8
30.5
19.2
59.7
36.7
60.2
38.5
84.8
41.4

70.8
63.4
35.8
38.0
58.8
20.8
67.0
41.2
81.8
53.0

Vertical Half
A
B
50.1
36.2
79.5
22.6
44.1
38.9
45.2
23.5
73.8
31.4

65.8
33.1
76.0
42.3
31.8
48.5
32.3
55.4
53.1
47.1

TABLE VI
C LASS P REDICTION A CCURACY FOR (A) A LL C ONV N ET BASELINE MODEL WITH DROPOUT AND (B) A LL C ONV N ET C ORRELATED P OISSON MODEL
WITH DROPOUT.
Class
airplane
automobile
bird
cat
deer
dog
frog
horse
ship
truck

Test Set
A

B

91.2
92.5
81.7
75.3
87.3
75.7
92.3
92.2
93.0
90.9

86.6
93.1
84.5
65.8
83.8
83.2
87.5
91.5
92.0
90.3

Central Occlusion
A
B
87.2
80.6
65.8
63.1
75.4
57.1
77.1
86.5
82.4
86.3

82.0
82.9
77.1
53.8
75.6
71.9
69.5
86.6
79.9
85.5

Checker Board
A
B

Horizontal Bars
A
B

81.7
38.0
56.3
55.6
80.7
48.7
85.5
61.2
79.4
63.9

85.1
9.5
8.3
10.7
51.3
10.9
0.9
24.5
58.6
18.6

75.4
49.5
69.8
45.2
78.9
62.6
73.8
68.2
82.8
74.4

across all classes with a mean performance increase on the
test set of ∼ 10%.
Independent Gaussian models improve performance over
the baseline model in all sets of images. This is expected,
as independent noise is similar to dropout. Hence, it can be
interpreted as adding dropout to the first layer of the network,
which can explain the performance benefit.
The correlated Gaussian models improved the performance
of the network for five out of the seven categories against
the baseline model with dropout. This is promising because
the addition of dropout into the model with correlated noise
increases performance on average.
The independent Poisson model with dropout improved performance against the baseline model with dropout on the test
set and six out of the seven occlusion sets. We hypothesized
earlier that an activity penalty with an independent Poisson
model can lead to strong classification performance, but it
seems that dropout along with Poisson noise seems to be a
good regularizer for the network.
The correlated Poisson models perform better than the baseline model on all the occluded image sets. The improvements
with the correlated model are consistent across different image
classes as shown in Table VI.
In general, the addition of correlated noise into the baseline
model with and without dropout increases recognition performance across the occlusion sets.

71.3
20.3
52.6
47.4
83.7
52.0
7.2
55.0
73.4
43.5

Vertical Bars
A
B
85.7
13.8
44.8
49.9
33.3
54.2
5.2
58.1
67.1
30.0

78.3
36.7
67.8
49.1
63.5
65.7
36.6
64.6
62.7
58.3

Horizontal Half
A
B
89.4
56.3
29.1
24.3
64.7
37.7
57.1
54.2
75.1
58.9

69.3
62.8
54.5
30.5
58.8
40.2
67.7
55.2
88.0
63.3

Vertical Half
A
B
79.8
19.8
77.0
50.9
27.1
19.7
17.4
24.5
78.5
10.3

76.4
34.2
76.7
62.5
59.2
43.8
38.5
46.7
64.9
50.4

V. C ONCLUSION
In this work, we proposed a model of a stochastic neural network whose activity values are correlated based on the spatial
spread and selectivity of the neurons. We tested different noise
models, as described in Section III-A, on the classification task
of occluded and non-occluded images.
A simple model of correlated variability, inspired by the
visual cortex, is modeled using the spatial distance between
units in an output feature map and the kernel weights that
produced them. The variability can be drawn from different
distributions of specified correlation; for example, Gaussian
and Poisson distributions. Depending on whether the sampling
function is differentiable, back-propagation can continue as
normal, or it will rely on an estimate of the gradient. There
are different methods to estimate the gradient, including reparameterization and straight-through estimation.
Preliminary results show that correlated variability added
to a single layer can perform better than other noise models.
In fact, in ten of the twelve occlusion cases we tested, both
with and without additional regularization, our best performing
models had some form of correlated noise. It remains to
be investigated whether this trend is robust across other
network architectures, types of occlusion, etc. It also remains
to be investigated how adding correlated noise to multiple
layers (as opposed to a single layer) affects the classification
performance of the network. However, sampling from high-

dimensional correlated distributions can be computationally
expensive. One of the main areas of focus in the future
will be how to balance computational tractability with the
performance benefits of incorporating correlation.
We considered Poisson distributions, as cortical neurons
display Poisson-like statistics in their spike arrival times.
Samples from a Poisson distribution can also approximate
dropout in the case when the rates are low. However, we
observed that the activations in the Poisson models are highly
variable. The standard deviation of Poisson noise scales with
the square root of the activation, so we expect that Poisson
noise is a weaker regularizer when the activations are higher.
This motivates the need for an activity penalty, in addition to
the Poisson noise. Our experiments show that dropout paired
with Poisson noise is also a strong regularizer, improving
recognition performance on the occluded image sets.
While we used a similar version of the straight-through
estimator for the Poisson models, there is a way to form a
continuous relaxation of a Poisson distribution using the work
in [8], [9]. It is also possible to use the idea of deconvolutions/fractional strided convolutions [1] to understand the
effect of correlated noise in input space. We intend to pursue
this as part of our future work.
ACKNOWLEDGEMENTS
We would like to thank Victor Reyes Osorio and Brittany
Reiche for their feedback on this work.
R EFERENCES
[1] M. D. Zeiler and R. Fergus, “Visualizing and Understanding Convolutional Networks,” ArXiv e-prints, Nov. 2013.
[2] D. L. K. Yamins, H. Hong, C. F. Cadieu, E. A. Solomon, D. Seibert,
and J. J. DiCarlo, “Performance-optimized hierarchical models predict
neural responses in higher visual cortex,” Proceedings of the National
Academy of Sciences, vol. 111, no. 23, pp. 8619–8624, 2014.
[3] E. Zohary, M. N. Shadlen, and W. T. Newsome, “Correlated neuronal
discharge rate and its implications for psychophysical performance,”
Nature, vol. 370, pp. 140–143, Jul 1994.
[4] X. Wang, A. Shrivastava, and A. Gupta, “A-fast-rcnn: Hard positive generation via adversary for object detection,” CoRR, vol. abs/1704.03414,
2017.
[5] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: A simple way to prevent neural networks from overfitting,” J. Mach. Learn. Res., vol. 15, no. 1, pp. 1929–1958, Jan. 2014.
[6] Y. Bengio, N. Léonard, and A. C. Courville, “Estimating or propagating gradients through stochastic neurons for conditional computation,”
CoRR, vol. abs/1308.3432, 2013.
[7] D. P. Kingma and M. Welling, “Auto-Encoding Variational Bayes,”
ArXiv e-prints, Dec. 2013.
[8] C. J. Maddison, A. Mnih, and Y. Whye Teh, “The Concrete Distribution:
A Continuous Relaxation of Discrete Random Variables,” ArXiv e-prints,
Nov. 2016.
[9] E. Jang, S. Gu, and B. Poole, “Categorical Reparameterization with
Gumbel-Softmax,” ArXiv e-prints, Nov. 2016.
[10] B. Poole, J. Sohl-Dickstein, and S. Ganguli, “Analyzing noise in
autoencoders and deep networks,” ArXiv e-prints, Jun. 2014.
[11] T. DeVries and G. W. Taylor, “Dataset Augmentation in Feature Space,”
ArXiv e-prints, Feb. 2017.
[12] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, “mixup: Beyond
Empirical Risk Minimization,” ArXiv e-prints, Oct. 2017.
[13] M. A. Smith and A. Kohn, “Spatial and temporal scales of neuronal
correlation in primary visual cortex,” Journal of Neuroscience, vol. 28,
no. 48, pp. 12 591–12 603, 2008.

[14] J. T. Springenberg, A. Dosovitskiy, T. Brox, and M. A. Riedmiller, “Striving for simplicity: The all convolutional net,” CoRR, vol.
abs/1412.6806, 2014.
[15] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S.
Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. J. Goodfellow,
A. Harp, G. Irving, M. Isard, Y. Jia, R. Józefowicz, L. Kaiser, M. Kudlur,
J. Levenberg, D. Mané, R. Monga, S. Moore, D. G. Murray, C. Olah,
M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. A. Tucker,
V. Vanhoucke, V. Vasudevan, F. B. Viégas, O. Vinyals, P. Warden,
M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng, “Tensorflow: Largescale machine learning on heterogeneous distributed systems,” CoRR,
vol. abs/1603.04467, 2016.

