Accepted as a workshop contribution at ICLR 2015

D ENOISING AUTOENCODER WITH MODULATED
LATERAL CONNECTIONS LEARNS INVARIANT
REPRESENTATIONS OF NATURAL IMAGES

arXiv:1412.7210v4 [cs.NE] 31 Mar 2015

Antti Rasmus and Tapani Raiko
Aalto University
Finland
{antti.rasmus,tapani.raiko}@aalto.fi
Harri Valpola
ZenRobotics Ltd.
Vilhonkatu 5 A, 00100 Helsinki, Finland
harri@zenrobotics.com

A BSTRACT
Suitable lateral connections between encoder and decoder are shown to allow
higher layers of a denoising autoencoder (dAE) to focus on invariant representations. In regular autoencoders, detailed information needs to be carried through
the highest layers but lateral connections from encoder to decoder relieve this
pressure. It is shown that abstract invariant features can be translated to detailed
reconstructions when invariant features are allowed to modulate the strength of
the lateral connection. Three dAE structures with modulated and additive lateral
connections, and without lateral connections were compared in experiments using
real-world images. The experiments verify that adding modulated lateral connections to the model 1) improves the accuracy of the probability model for inputs, as
measured by denoising performance; 2) results in representations whose degree of
invariance grows faster towards the higher layers; and 3) supports the formation
of diverse invariant poolings.

1

I NTRODUCTION

Denoising autoencoders (dAE; Vincent et al., 2008) provide an easily accessible method for unsupervised learning of representations since training can be based on simple back-propagation and a
quadratic error function. An autoencoder is built from two mappings: an encoder that maps corrupted input data to features, and a decoder that maps the features back to denoised data as output.
Thus, in its basic form, autoencoders need to store all the details about the input in its representation.
Deep learning has used unsupervised pretraining (see Bengio, 2013, for a review) but recently purely
supervised learning has become the dominant approach at least in cases where a large number of
labeled data is available (e.g., Ciresan et al., 2010; Krizhevsky et al., 2012).
One difficulty with combining autoencoders with supervised learning is that autoencoders try to
retain all the information whereas supervised learning typically loses some. For instance, in classification of images, spatial pooling of activations throws away some of the location details while
retaining identity details. In that sense, the unsupervised and supervised training are pulling the
model in very different directions.
From a theoretical perspective, it is clear that unsupervised learning must be helpful at least in a
semi-supervised setting. Indeed, Kingma et al. (2014) obtained very promising results with variational autoencoders. This raises hopes that the same could be achieved with simpler dAEs.
Recently, Valpola (2015) proposed a variant of the denoising autoencoder that can lose information. The novelty is in lateral connections that allow higher levels of an autoencoder to focus on
invariant abstract features and in layer-wise cost function terms that allow the network to learn deep
1

Accepted as a workshop contribution at ICLR 2015

f (2)

g (1)

h(1)

h(1)

x̂

(a) Basic autoencoder

↵ = 0.25

ĥ(1)
f (1)

g (0)

x̃

ĥ(2)
g (1)

f (2)

ĥ(1)
f (1)

g (2)

h(2)

h(2)

g (0)

x̃

↵ = 4.0
x̂

(b) Lateral connections

(c) Layer size ratio

Figure 1: Examples of two-hidden-layer models: (a) denoising autoencoder and (b) Ladder network
with lateral connections. Illustration of ratio, α, of h(2) size to h(1) size (right) with hidden layers
of sizes 2 and 8. Ratio of α = 1.0 corresponds to hidden layers of equal size (a and b).
hierarchies efficiently. Valpola (2015) hypothesized that modulated lateral connections support the
development of invariant features and provided initial results with artificial data to back up the idea.
As seen in Figure 1, information can flow from the input to the output through alternative routes, and
the details no longer need to be stored in the abstract representation. This is a step closer to being
compatible with supervised learning which can select which types of invariances and abstractions
are relevant for the task at hand.
In this paper, we focus on investigating the effects of lateral connections. We extend earlier results
with experiments using natural image data and make comparisons with regular denoising autoencoders without lateral connections in Section 2. We show the following:
• The proposed structure attains a better model of the data as measured by ability to denoise. There are good reasons to believe that this indicates that the network has captured
a more accurate probabilistic model of the data since denoising is one way of representing
distributions (Bengio et al., 2013).
• Including the modulated lateral connections changes the optimal balance between the sizes
of layers from balanced to bottom heavy.
• The degree of invariance of the representations grows towards the higher levels in all tested
models but much faster with modulated lateral connections. In particular, the higher levels
of the model seem to focus entirely on invariant representations whereas the higher levels of
the regular autoencoder have a few invariant features mixed with a large number of details.
• Modulated lateral connections guide the layer above them to learn various types of poolings. The pooled neurons participate in several qualitatively different poolings that are each
selective and invariant to different aspects of the input.
1.1

I NVARIANT FEATURES THROUGH POOLING

There are typically many sources of variation which are irrelevant for classification. For example,
in object recognition from images, these sources could include position, orientation and scale of
the recognized object and illumination conditions. In order to correctly classify new samples, these
sources of variation are disregarded while retaining information needed for discriminating between
different classes. In other words, classification needs to be invariant to the irrelevant transformations.
A simple but naive way to achieve such invariance would be to list all the possible realizations of
objects under various transformations, but this is not usually practical due to the vast amounts of
possible realizations. Also, this does not offer any generalization to new objects.
A useful solution is to split the generation of invariance into more manageable parts by representing
the inputs in terms of features which are each invariant to some types of transformations. Since
different kinds of objects can be represented by the same set of features, this approach makes it
possible to generalize the invariances to new, unseen objects.
So rather than listing all the possible realizations of individual objects, we can list possible realizations of their constituent features. Invariance is achieved by retaining only the information about
2

Accepted as a workshop contribution at ICLR 2015

whether any of the possible realizations is present and discarding information about which one exactly. Such an operation is known as pooling and there are various ways of implementing it. In the
case of binary inputs, OR operation is a natural choice. There are many ways to generalize this to
continuous variables, including maximum operation (Riesenhuber & Poggio, 1999) or summation
followed by a concave function (Fukushima, 1979).
While pooling achieves invariance to irrelevant transformations, the features also need to be discriminative along relevant dimensions. This selectivity is often achieved by coincidence detection, i.e. by
detecting the simultaneous presence of multiple input features. In the binary case, it can be implemented by an AND operation. In the continuous case, possibilities include extensions of the AND
operation, such as product or summation followed by a convex function, but also lateral inhibition
(i.e., competition) among a set of feature detectors. All of these operations typically produce sparse
coding where the output features are sensitive to specific combinations of input features. Since this
type of coding tends to increase the number of features, it is also known as feature expansion.
The idea of alternating pooling and feature expansion dates at least back to Hubel & Wiesel (1962)
who found that the early stages of visual processing in the cat cerebral cortex have alternating
steps of feature expansion, implemented by lateral competition among so called simple cells, and
invariance-generating poolings by so called complex cells. In such hierarchies of alternating steps,
the degree of invariance grows towards the higher levels. Cortical processing also includes various
normalizations, a feature which has also been included in some models (e.g., Fukushima, 1979).
There are various ways of finding poolings that generate invariances (from specialized to general):
1. Invariance by design. For instance, invariance to translation and small deformations is
achieved by pooling shifted versions of a feature (Fukushima, 1979). Similar pooling operations are now popular in convolutional neural networks (see, e.g., Schmidhuber, 2015).
2. Invariance to hand-crafted transformations. The transformations are applied to input samples (e.g., image patches can be shifted, rotated, scaled or skewed, and colors or contrast
can be modified) and pooling is then learned by requiring the output to stay constant over
the transformation. This category includes supervised learning from inputs deformed by
various transformations.
3. Invariance to transformations over time. Relies on nature to provide transformations as
sequences during which the underlying feature (e.g., identity of object) changes slower
than the transformation (e.g., Földiák, 1991).
4. Invariance by exploiting higher-order correlations within individual samples. This is how
supervised learning can find poolings: target labels correlate nonlinearly with inputs. There
are also unsupervised methods that can do the same. For example, subspace ICA can find
complex-cell like poolings for natural images (Hyvärinen & Hoyer, 2000).
We focus on the last type: exploiting higher-order correlations. Very few assumptions are made so
the method is very general but it is also possible to combine this approach with supervised learning
or any of the more specialized ways of generating invariances.
1.2

D ENOISING AUTOENCODERS

Autoencoder networks have a natural propensity to conserve information and are therefore well
suited for feature expansion. Autoencoder networks consist of two parameterized functions, encoding f and decoding g. Function f maps from input space x to feature space h, and g in turn maps
back to input space producing a reconstruction, x̂, of the original input, when the training criterion
is to minimize the reconstruction error. This enables learning of features in an unsupervised manner.
Denoising autoencoder (Vincent et al., 2008) is a variant of the traditional autoencoder, where the
input x is corrupted with noise and the objective of the network is to reconstruct the original uncorrupted input x from the corrupted x̃. Bengio et al. (2013) show that denoising autoencoders
implicitly estimate the data distribution as the asymptotic distribution of the Markov chain that
alternates between corruption and denoising. This interpretation provides a solid probabilistic foundation for them. Consequently, the denoising teaching criterion enables learning of over-complete
representations, a property which is crucial for adding lateral connections to an autoencoder.
3

Accepted as a workshop contribution at ICLR 2015

As with normal feedforward networks, there are various options for choosing the cost function but,
in the case of continuous input variables, a simple choice is
C = kx̂ − xk2 = kg(f (x̃)) − xk2 .

(1)

Denoising autoencoders can be used to build deep architectures either by stacking several on top of
each other and training them in a greedy layer-wise manner (e.g., Bengio et al., 2007) or by chaining
several encoding and decoding functions and training all layers simultaneously. For L layers and
encoding functions f (l) , the encoding path would compose as f = f (L) ◦ f (L−1) ◦ . . . f (1) .
We denote the intermediate feature vectors by h(l) and corresponding decoded, denoised, vectors by
ĥ(l) . Figure 1a depicts such a structure for L = 2. Encoding functions are of the form
(l)
(l) 
h(l) = f (l) (h(l−1) ) = φ Wf h(l−1) + bf , 1 ≤ l ≤ L,
(2)
starting from h(0) = x̃. The corresponding decoding functions are
ĥ(L) = h(L)
(l)

ĥ

(3)

(l)

(l+1)

(0)

(1)

= g (ĥ

x̂ = g

(ĥ

)=φ

)=

Wg(l) ĥ(l+1)

Wg(0) ĥ(1)

+

+

b(l)
g



,

1 ≤ l ≤ L − 1,

b(0)
g .

(4)
(5)

Function φ is the activation function and typically left out from the lowest layer.

2

E XPERIMENTS

The tendency of regular autoencoders to preserve information seems to be at odds with the development of invariant features which relies on poolings that selectively discard some types of
information. Our goal here is to investigate the hypothesis that suitable lateral connections allow
autoencoders to discard details from higher layers and only retain abstract invariant features because
the decoding functions g (l) can recover the discarded details from the encoder.
We compare three different denoising autoencoder structures: basic denoising autoencoder and two
variants with lateral connections. We experimented with various model definitions prior to deciding
the ones defined in Section 2.1 because there are multiple ways to merge lateral connections.
2.1

M ODELS WITH L ATERAL C ONNECTIONS

We add lateral connections from h(l) to ĥ(l) as seen in Figure 1b. Autoencoders trained without
noise would short-circuit the input and output with an identity mapping. Now that input contains
noise, there is pressure to find meaningful higher-level representations that capture regularities and
allow for denoising. Note that the encoding function f has the same form as before in Eq. (2).
2.1.1

A DDITIVE LATERAL CONNECTIONS

As the first version, we replace the decoding functions in Eq. (3–4) with
(L) 
ĥ(L) = g (L) (h(L) ) = (h(L) + b(L)
σ a(L) h(L) + bb ,
(6)
a )


(l)
ĥ(l) = g (l) (h(l) , ĥ(l+1) ) = (h(l) + b(l)
σ a(l) h(l) + bb + φ Wg(l) ĥ(l+1) + b(l)
, (7)
a )
g
(l)

(l)

where is the element-wise (i.e., Hadamard) product, a(l) , ba , and bb are learnable parameter
vectors along with weights and biases, and σ is a sigmoid function to ensure that the modulation
stays within reasonable bounds. Function g (0) stays affine as in Eq. (5). The functional form of
Eq. (6), with element-wise decoding, is motivated by element-wise denoising functions that are used
in denoising source separation (Särelä & Valpola, 2005) and corresponds to assuming the elements
of h independent a priori.
4

Accepted as a workshop contribution at ICLR 2015

2.1.2

M ODULATED LATERAL CONNECTIONS

Our hypothesis is that an autoencoder can learn invariances efficiently only if its decoder can make
good use of them. Valpola (2015) proposed connecting the top-down mapping inside the sigmoid
term of Eq. (7), a choice motivated by optimal denoising in hierarchical variance models.
Our final proposed model includes the encoding functions in Eq. (2), the top connection g (L) in Eq.
(6), bottom decoding function g (0) as in Eq. (5), but the middle decoding functions are defined as
(l) 
g (l) (h(l) , ĥ(l+1) ) = (h(l) + b(l)
σ a(l) h(l) + Wg(l) ĥ(l+1) + bb , 1 ≤ l ≤ L − 1. (8)
a )
In contrast to additive lateral connection in Eq. (7), the signal from the abstract layer ĥ(l+1) is used
to modulate the lateral connection so that the top-down connection has moved from φ(·) to σ(·), and
(l)
(l)
the bias bg has dropped because it is redundant with the bias term bb in σ(·).
Modulated (also known as gated or three-way) connections have been used in autoencoders before
but in a rather different context. Memisevic (2011) uses a weight tensor to connect two inputs to a
feature. We connect the ith component of h(l) only to the ith component of ĥ(l) , keeping the number
of additional parameters small.
2.2

T RAINING PROCEDURE

In order to compare the models, we optimized each model structure constraining to 1 million parameters and 1 million mini-batch updates to find the best denoising performance, that is, the lowest
reconstruction cost C in Eq. (1). The better the denoising performs the better the implicit probabilistic model is. The tasks is the same for all models, so the comparison is fair. With two-layer models
(L = 2) the focus was to find the optimal size for layers, especially the ratio of h(2) size to h(1) size
(see Fig. 1c for the definition of the ratio α). All the models use rectified linear unit as the activation
function φ(·) and the noise was Gaussian with zero mean and standard deviation scaled to be 50%
of that of the data.
In order to find the best possible baseline for comparison, we evaluated weight tying for autoencoder
(l)

(l+1) T

, and noticed that tying weights resulted in
without lateral connections , that is, Wg = Wf
faster convergence and thus better denoising performance than without weight tying. However, when
combining both so that weights are tied in the beginning and untied for the latter half of the training,
denoising performance improved slightly (by 1%), but did not affect the relative difference of various
models. Since weight tying had only negligible impact on the results, but it complicates training
and reproducibility, we designed the experiments so that all models use tied weights counting tied
weights as separate parameters. Results of the denoising performance are described in Section 2.3.
We performed the analysis on natural images because the invariances and learned features are easy
to visualize, we know beforehand that such invariances do exist, and because computer vision is
an important application field in its own right. We used 16 × 16 patches from two image datasets:
CIFAR-10 (Krizhevsky & Hinton, 2009) and natural images used by Olshausen & Field (1996)1 .
We refer to this dataset as O&F.
The training length was limited to 1 million mini-batch updates with mini-batch of size 50 and learning rate was adapted with ADADELTA (Zeiler, 2012). The best variants of each model were then
trained longer, for 4 million updates, and further analyzed to determine invariance of the learned representations. This is described and reported in Section 2.4. Supplementary material provides more
details about data preprocessing, division between training and validation sets, training procedure,
hyperparameters used with ADADELTA, and how weights were initialized.
We also tried stacked layer-wise training by training each layer for 500,000 updates followed by finetuning phase of another 500,000 updates such that the total number of updates for each parameter
equals to 1 million. We also tried a local cost function and noise source on each layer or using
the global cost, but we did not find any such stacked training beneficial compared to the direct
simultaneous training of all layers, which is what we report in this paper.
1

Available at https://redwood.berkeley.edu/bruno/sparsenet/

5

Accepted as a workshop contribution at ICLR 2015

Mod

No-lat

Add

Mod

Cost

No-lat

Add

0.16

0.115

0.18

0.120

0.125

0.20

Cost

0.0

0.1

0.2

0.3

0.4

0.5

1.0

2.0 3.0

0.0

0.1

0.2

0.3

α

0.4

0.5

1.0

2.0 3.0

α

Figure 2: The best validation cost per element as a function of α, the ratio of h(2) size to h(1) size, for
CIFAR-10 (left) and O&F (right) datasets. Dotted line is the result of linear denoising (model Linear
in Table 1), two dashed lines represent the denoising performance of one-layer models (L = 1)
with and without lateral connections according to their colors. Note that Add and Mod models are
identical when L = 1. The scale of the horizontal axis is linear until 0.5 and logarithmic after that.
(2)

γi

Mod 256-1622-50

Add 256-839-336

No-lat 256-590-589

1.0
0.8
0.6
0.4
0.2
0.0
10−2

10−1

100

10−2

10−1

100

10−2

10−1

100

Figure 3: Translation invariance measure of neurons on h(2) as a function of significance. Color
indicates the average sign of the connected weights from negative (blue) to positive (red). Best
viewed in color. See Section 2.4 for more details.

2.3

D ENOISING PERFORMANCE

The results of denoising performance for models with one and two layers is presented in the Figure 2
which shows the lowest reconstruction cost on a validation dataset for both datasets. Each configuration was trained 5 times with different random initialization for confidence bounds calculated as
corrected sample standard deviation of the lowest reconstruction cost.
The best performing No-lat model (autoencoder without lateral connections) is a one-layer model
and is shown in dashed line. The best two-layer No-lat models in CIFAR-10 and O&F have ratios
of αmin = 1.0 and αmin = 1.4, respectively. Since No-lat autoencoders need to push all the
information through each layer, it is intuitive that very narrow bottlenecks are bad for the model,
that is, large and small ratios perform poorly. A further study of why the optimal ratio is larger for
O&F revealed that the lower layer can be smaller because the effective dimensionality in terms of
principal components is lower for O&F compared to CIFAR-10. Second layer is not beneficial with
No-lat model given the parameter number constraint.
Mod (modulated lateral connections) model benefits from the second layer and works best when
the ratio is small, namely αmin = 0.03 and αmin = 0.12 for CIFAR-10 and O&F, respectively.
The second layer does not hurt or benefit Add (additive lateral connection) model significantly and
its performance is between No-lat and Mod models. The results are also presented as numbers in
Table 1 in supplementary material.
6

Accepted as a workshop contribution at ICLR 2015

2.4

R ESULTING INVARIANT FEATURES

Practically no prior information about poolings was incorporated in either the model structure or
treatment of training data. This means that any invariances learned by the model must have been
present in the higher-order correlations of the inputs, as explained in Section 1.1. It is well known
that such invariances can be developed from natural images (e.g., Hyvärinen & Hoyer, 2000) but the
question is, how well are the different model structures able to represent and learn these features.
To test this, we generated sets of rotated, scaled and translated images and measured how invariant
the activations h(l) are for each type of transformation separately. As an example for translation,
each set contained 16 translated images (from a 4 × 4 grid). For each set s in a given transformation
type, we calculated the mean activation hh(l) is and compared their variances with the variance
var{h(l) } over all samples:
(l)
(l)
(l)
γi = var{hhi is }/var{hi } .
(9)
(l)

From the definition it follows that 0 ≤ γi ≤ 1. If the feature is completely invariant with respect
(l)
to the transformation, γi equals one2 . As the overall conclusions are similar to all tested transformations, the main text concentrates on the translation invariance. Results for other invariances are
(l)
reported in supplementary material, Section 4.4. The average layer-wise invariance, γ (l) = hγi ii ,
grows towards the higher layers in all models but much faster in the best Mod models than in others3 ,
i.e. for CIFAR-10 the best Mod model has γ (0)...(2) = (0.20, 0.31, 0.84), whereas for the best No-lat
γ (0)...(2) = (0.20, 0.23, 0.30). All γ (l) values are reported in Table 1 in supplementary material.
(2)

To further illustrate this, we plotted in Figure 3 the invariance measures γi for the best variant of
(2)
each model. In each plot, dots correspond to hidden neurons hi and the color reflects the average
sign of the encoder connection to that neuron (blue for negative, red for positive). The horizontal
axis is the significance of a neuron, a measure of how much the model uses that hidden neuron,
which is defined and analyzed in supplementary material, Section 4.3.
(2)

There are two notable observations. First, all hi neurons of Mod model are highly invariant,
whereas the other models have only few invariant neurons and the vast majority of neurons have
very low invariance. For No-lat model, invariance seems to be even smaller for those neurons that
the model uses more. Moreover, we tested that the second layer of Mod model stays highly invariant
even if the layer size is increased (shown in Figure 7c in supplementary material). Second, invariant
neurons tend to have far more and stronger negative than positive weights, especially so with Mod
and No-lat models. Since the nonlinearity φ(x) on each layer was the rectified linear unit, a convex
function which saturates on the negative side, negative tied weights mean that the network flipped
these functions into −φ(−x), that is, concave functions that saturate on the positive side resembling
OR operation. This interpretation is further discussed in supplementary material, Section 4.3.2.
2.5

L EARNED POOLINGS

The modulated Mod model used practically all of the second layer neurons for pooling. When studying the poolings, we found that typically every Layer 1 neuron participates in several qualitatively
different poolings. As can be seen from the Figure 4, each Layer 1 neuron (shown on the left-most
column) participates in different kinds of poolings each of which is sensitive to a particular set of
features and invariant to other types. For example, Layer 1 neuron (c) is selective to orientation,
frequency and color but it participates in three different Layer 2 poolings . The first one is selective
to color but invariant to orientation. The second one is selective to orientation but invariant to color.
The third one only responds to high frequency and orientation. More details of the analysis are
available in supplementary material.
2
Various measures of invariance have been proposed (e.g., Goodfellow et al., 2009). Our invariance measure
γ is closely related to autocorrelation which has been used to measure invariance, e.g., by Dosovitskiy et al.
(2014). The set of translated patches includes pairs with various translations and therefore our measure is a
weighted average of the autocorrelation function. The wider the autocorrelation, the larger the measure γ.
3
Our measure γ can be fooled by copying the same invariant feature to many hidden neurons but we verified
that this is not happening here: slow feature analysis is robust against such redundancy but yields qualitatively
the same results for the tested networks.

7

Accepted as a workshop contribution at ICLR 2015

(a)

(b)

(c)

(d)

Figure 4: Various pooling groups a neuron. Each set (a)-(d) represents three relevant pooling groups
(1)
(1)
the selected neuron ha . . . hd , depicted in the first column, belongs to. Each row represents an
(2)
h pooling neuron by showing the 20 most relevant h(1) neurons associated with it. Poolings
were found by selecting a few Layer 1 neurons and following their strongest links to Layer 2 to
identify the poolings in which they participate. Consecutively, the Layer 1 neurons corresponding to
each pooling group were identified by walking back the strongest links from each Layer 2 neuron.
Best viewed in color. The procedure of choosing features in the plot is also depicted in Figure 8 in
supplementary material.

3

D ISCUSSION

The experiments showed that invariance in denoising autoencoders increased towards the higher layers but significantly so only if the decoder had a suitable structure where details could be combined
with invariant features via multiplicative interactions. Lateral connections from encoder to decoder
allowed the models to discard details from the higher levels but only if the details and invariant
features were combined suitably.
The best models with modulated lateral connections were able to learn a large number of poolings in
an unsupervised manner. We tested their invariance but nothing in the model biased learning in that
direction and we observed invariance and selective discrimination of several different dimensions
such as color, orientation and frequency.
In summary, these findings are fully in line with the earlier proposition that the unsupervised denoising autoencoder with modulated lateral connections can work in tandem with supervised learning
because, as we have shown here for the first time, the higher layers of the model have the ability
to focus on abstract representations and, unlike regular autoencoders, should therefore be able to
discard details if supervised learning deems them irrelevant. It now becomes possible to combine
autoencoders with the popular supervised learning pipelines which include in-built pooling operations (see, e.g., Schmidhuber, 2015).
There are multiple ways to extend the work, including 1) explicit bias towards invariances; 2) sparse
structure such as convolutional networks, making much larger scale models and deeper hierarchies
feasible; 3) dynamic models; and 4) semi-supervised learning.

R EFERENCES
Bengio, Y, Lamblin, P, Popovici, D, and Larochelle, H. Greedy layer-wise training of deep networks.
In Advances in Neural Information Processing Systems (NIPS 2006), 2007.
8

Accepted as a workshop contribution at ICLR 2015

Bengio, Y. Deep learning of representations: Looking forward. In Statistical Language and Speech
Processing, pp. 1–37. Springer, 2013.
Bengio, Y, Yao, L, Alain, G, and Vincent, P. Generalized denoising auto-encoders as generative
models. In Advances in Neural Information Processing Systems, pp. 899–907, 2013.
Ciresan, D. C, Meier, U, Gambardella, L. M, and Schmidhuber, J. Deep big simple neural nets for
handwritten digit recogntion. Neural Computation, 22(12):3207–3220, 2010.
Dosovitskiy, A, Springenberg, J. T, Riedmiller, M, and Brox, T. Discriminative unsupervised feature
learning with convolutional neural networks. In Ghahramani, Z, Welling, M, Cortes, C, Lawrence,
N, and Weinberger, K (eds.), Advances in Neural Information Processing Systems (NIPS 2014),
pp. 766–774. 2014.
Földiák, P. Learning invariance from transformation sequences. Neural Computation, 3:194–200,
1991.
Fukushima, K. Neural network model for a mechanism of pattern recognition unaffected by shift in
position - Neocognitron. Trans. IECE, J62-A(10):658–665, 1979.
Goodfellow, I, Lee, H, Le, Q. V, Saxe, A, and Ng, A. Y. Measuring invariances in deep networks.
In Advances in Neural Information Processing Systems (NIPS), pp. 646–654, 2009.
Hubel, D. H and Wiesel, T. N. Receptive fields, binocular interaction and functional architecture in
the cat’s visual cortex. The Journal of physiology, 160(1):106, 1962.
Hyvärinen, A and Hoyer, P. Emergence of phase-and shift-invariant features by decomposition of
natural images into independent feature subspaces. Neural computation, 12(7):1705–1720, 2000.
Kingma, D. P, Mohamed, S, Rezende, D. J, and Welling, M. Semi-supervised learning with deep
generative models. In Ghahramani, Z, Welling, M, Cortes, C, Lawrence, N, and Weinberger, K
(eds.), Advances in Neural Information Processing Systems (NIPS 2014), pp. 3581–3589. Curran
Associates, Inc., 2014.
Krizhevsky, A and Hinton, G. Learning multiple layers of features from tiny images. Technical
report, University of Toronto, 2009.
Krizhevsky, A, Sutskever, I, and Hinton, G. E. ImageNet classification with deep convolutional
neural networks. In Advances in Neural Information Processing Systems (NIPS 2012), pp. 1106–
1114, 2012.
Memisevic, R. Gradient-based learning of higher-order image features. In 2011 IEEE International
Conference on Computer Vision (ICCV), pp. 1591–1598, November 2011.
Olshausen, B. A and Field, D. J. Emergence of simple-cell receptive field properties by learning a
sparse code for natural images. Nature, 381:607–609, 1996.
Raiko, T, Valpola, H, and LeCun, Y. Deep learning made easier by linear transformations in perceptrons. In Lawrence, N. D and Girolami, M (eds.), AISTATS, volume 22 of JMLR Proceedings, pp.
924–932. JMLR.org, 2012.
Riesenhuber, M and Poggio, T. Hierarchical models of object recognition in cortex. Nature Neuroscience, 2(11):1019–1025, 1999.
Särelä, J and Valpola, H. Denoising source separation. Journal of Machine Learning Research, 6:
233–272, 2005.
Schmidhuber, J. Deep learning in neural networks: an overview. Neural Networks, 61:85–117,
2015.
Valpola, H. From neural PCA to deep unsupervised learning. In Advances in Independent Component Analysis and Learning Machines. Elsevier, 2015. Preprint available as arXiv:1411.7783.
Vincent, P, Larochelle, H, Bengio, Y, and Manzagol, P.-A. Extracting and composing robust features
with denoising autoencoders. Technical Report 1316, Université de Montréal, dept. IRO, 2008.
Zeiler, M. D. Adadelta: An adaptive learning rate method. arXiv preprint arXiv:1212.5701, 2012.
9

Accepted as a workshop contribution at ICLR 2015

Table 1: Denoising performance and translation invariance measure of selected models. All models
have exactly the same input layer and data so the average invariance is the same for all models, e.g.
for CIFAR-10, γ (0) = 0.20.
Dataset
CIFAR-10
CIFAR-10
CIFAR-10
CIFAR-10
CIFAR-10
CIFAR-10
O&F
O&F
O&F
O&F
O&F
O&F

4

Model
Linear
No-lat
Add/Mod
No-lat
Add
Mod
Linear
No-lat
Add/Mod
No-lat
Add
Mod

Layer sizes
256
256-1948
256-1937
256-590-589
256-839-336
256-1622-50
256
256-1948
256-1937
256-512-718
256-1061-212
256-1395-100

Ratio α
0.00
0.00
1.00
0.40
0.03
0.00
0.00
1.40
0.20
0.07

Min cost ± std
0.20316 ± 0.00011
0.16144 ± 0.00007
0.15793 ± 0.00007
0.17595 ± 0.00034
0.15734 ± 0.00015
0.15086 ± 0.00022
0.14686 ± 0.00010
0.12290 ± 0.00008
0.11891 ± 0.00006
0.12446 ± 0.00015
0.11723 ± 0.00009
0.11234 ± 0.00005

Invariance, γ (l)
NA
0.20, 0.29
0.20, 0.29
0.20, 0.23, 0.30
0.20, 0.27, 0.33
0.20, 0.31, 0.84
NA
0.16, 0.24
0.16, 0.25
0.16, 0.19, 0.27
0.16, 0.25, 0.41
0.16, 0.24, 0.95

S UPPLEMENTARY MATERIAL

4.1

P REPROCESSING OF DATA

Patches of size 16 × 16 were sampled randomly and continuously during training. Separate test
images were put aside for testing generalization performance: the last 10,000 samples for CIFAR10 and the sixth image of O&F dataset. Continuous sampling allows generation of millions of data
samples alleviating overfitting problems. O&F dataset was already (partially) whitened so no further
preprocessing was applied to it. RGB color patches of CIFAR-10 were whitened and dimensionality
was reduced to 256 with PCA to match the dimensionality of grayscale images of O&F. Despite
dimensionality reduction, 99% of the variance was retained.
4.2

D ETAILS OF THE TRAINING PROCEDURE

White additive Gaussian noise of σN = 0.5 was used for corrupting the inputs which were scaled
to have standard deviation of σ = 1.0. During training, ADADELTA was used to adapt the learning
rate with its momentum set to 0.99, and  = 10−8 . All weight vectors were initialized from normal
distribution to have a norm of 1.0, and orthogonalized. In order to improve the convergence speed
of all models, we centered the hidden unit activations following Raiko et al. (2012): there is an
auxiliary bias term β (Raiko et al., 2012, Eq. (2)), applied immediately after the nonlinearity, that
centers the output to zero mean.4
4.3

A NALYSIS OF MAPPINGS

We analyzed the mappings learned by different types of models in several ways and present here
some of the most interesting findings.
First, it turned out that different models had very different proportions of invariant neurons on h(2)
(invariance measure γ is defined in Eq. (9)) and we wanted to understand better what was going on.
Some key questions were how important roles different types of features had and how the invariances
were formed. Second level invariances could be low-frequency features which are invariant already
on the first layer (and thus not particularly interesting) or formed through pooling Layer 1 neurons.
The first question can be answered by looking at where the connections are coming from. The
(l)
connections Wg are visualized in Figures 5a–5c. The neurons on each layer have been ordered
with respect to invariance which increases from left to right. The connecting edges have been colored
according to the sign of the connecting weight and the strength of each edge reflects the significance
of the connection. Significance is defined as the proportion of variance that the higher-level neuron
4

We did not use the other transformation α since it would have required shortcut connections.

10

Accepted as a workshop contribution at ICLR 2015

generates on the lower-level neurons. We initially tried visualizing simply the magnitudes of the
weights but the problem is that when an input neuron has a low variance or the output neuron is
saturated, a large weight magnitude does not indicate that the connection is important; it would not
make much difference if such a connection were removed.
4.3.1

S IGNIFICANCE MEASURE
(l)

When visualizing Wg , we first took the squares and scaled them by the input neuron’s variance
(l+1)
var{hi
}. Assuming input neurons are independent, this quantity reflects the input variance each
(l)
neuron hj receives. Depending on the saturation of the neuron, a smaller or greater proportion
(l)

of this variance is transmitted to the actual output variance var{hj }. We therefore scaled all the
(l)

(l)

incoming variances for each hj such that their sum matches the output variance var{hj }. We
named this quantity the significance of the connection and it approximately measures where the
output variance of each layer originates from. This significance is also depicted in Figure 3 where
(2)
the x coordinate is the sum of output significances for each hi .
4.3.2

ROLE OF NEGATIVE WEIGHTS

It turned out that the invariant neurons tend to have far more and stronger negative than positive
weights. We visualized this with color in Figure 5: blue signifies negative and red positive weights.
In the images, the connections are translucent which means that equal number (and significance) of
positive and negative weights results in purple color.
A striking feature of these plots is that the most invariant features tend to have all negative weights.
Since the nonlinearity φ(x) on each layer was the rectified linear unit, a convex function which
saturates on the negative side, negative tied weights mean that the network flipped these functions
into −φ(−x), that is, concave functions that saturate on the positive side. It therefore looks like the
network learned to add a layer of concave OR-like invariance-generating poolings on top of convex
AND-like coincidence detectors. Forming convex AND using positive weights and concave OR
using negative weights is demonstrated respectively as truth tables in Table 2 and Table 3, and the
geometric form is illustrated in Figure 6a and Figure 6b.
4.4

I NVARIANCE

We also studied invariance of the second layer neurons for scaling and rotation transformations
using the same invariance measure as defined in Section 2.4 for translation. We formed sets of 16
samples by scaling CIFAR-10 images with a zoom factors 0.6, 0.65, . . . , 1.35 and rotating images
−32◦ , −28◦ , . . . , 28◦ for scaling and rotation invariance experiments, respectively. The results are
shown in Figure 7a and Figure 7b and are similar to the translation invariance in Figure 3.
Figure 7c illustrates the impact of increasing α, i.e. increasing the size of the second layer. Notably
all second layer neurons stay highly invariant even when the layer size is increased.

11

Accepted as a workshop contribution at ICLR 2015

(a) Connections of Mod model 256-1622-50

(b) Connections of Add model 256-839-336.

(c) Connections of No-lat model 256-590-589.

Figure 5: Neurons are ordered according to increasing translation invariance from left to right.
(l)
Blue denotes negative and red positive weights in Wg . Strength of the connections depends on the
significance of the connection (see text for details). Layer 2 neurons of this model are also visualized
in Figure 3. Best viewed in color.
12

Accepted as a workshop contribution at ICLR 2015

Table 2: Truth table for logical AND operation given three-element binary input vector x and a
single perceptron with ReLU activation function φ(·).
φ(z) = φ(xT w + b) = φ(x0 w0 + x1 w1 + x2 w2 + b)
x0
0
0
0
0
1
1
1
1

x1
0
0
1
1
0
0
1
1

x2
0
1
0
1
0
1
0
1

wi
1
1
1
1
1
1
1
1

b
-2
-2
-2
-2
-2
-2
-2
-2

z
-2
-1
-1
0
-1
0
0
1

φ(z)
0
0
0
0
0
0
0
1

AND
0
0
0
0
0
0
0
1

Table 3: Truth table for logical OR operation given three-element binary input vector x and a single
perceptron with ReLU activation function φ(·).
φ(z) = φ(xT w + b) = φ(x0 w0 + x1 w1 + x2 w2 + b)
x0
0
0
0
0
1
1
1
1

x1
0
0
1
1
0
0
1
1

x2
0
1
0
1
0
1
0
1

wi
-1
-1
-1
-1
-1
-1
-1
-1

b
1
1
1
1
1
1
1
1

z
1
0
0
-1
0
-1
-1
-2

φ(z)
1
0
0
0
0
0
0
0

Convex AND function

2

NOR
1
0
0
0
0
0
0
0

OR
0
1
1
1
1
1
1
1

Concave OR function

2

1

1 − φ(z)
0
1
1
1
1
1
1
1

1

0

0
0

1

2

3

0

Number of active inputs

1

2

3

Number of active inputs

(a) Function φ(xT w + b), wi = 1, b = −2

(b) Function 1 − φ(xT w + b), wi = −1, b = 1

Figure 6: Conceptual illustration of linear rectifier unit, φ(·), performing logical AND and OR
operations for three-element binary input x, xi ∈ {0, 1} after the affine transform z = xT w + b =
x0 w0 + x1 w1 + x2 w2 + b. AND operations has positive weights and convex form, whereas OR
operation has negative weights and concave functional form.

13

Accepted as a workshop contribution at ICLR 2015

Mod 256-1622-50

(2)

γi

Add 256-839-336

No-lat 256-590-589

1.0
0.8
0.6
0.4
0.2
0.0
10−2

10−1

100

10−2

10−1

100

10−2

10−1

100

(a) Rotation invariance

Mod 256-1622-50

(2)

γi

Add 256-839-336

No-lat 256-590-589

1.0
0.8
0.6
0.4
0.2
0.0
10−2

10−1

100

10−2

10−1

100

10−2

10−1

100

(b) Scaling invariance

Mod 256-1622-50

(2)

γi

Mod 256-1500-75

Mod 256-1395-100

1.0
0.8
0.6
0.4
0.2
0.0
10−3

10−2

10−1

100

10−3

10−2

10−1

100

10−3

10−2

10−1

100

(c) Translation invariance for Mod models with larger α

Figure 7: Various invariance measures for neurons of h(2) as a function of significance for CIFAR10. Color indicates the average sign of the connected weights from negative (blue) to positive (red).
Best viewed in color. See Section 2.4 for more details.

14

Accepted as a workshop contribution at ICLR 2015

Figure 8: Method for selecting pooling groups including a given h(1) neuron contains two phases.
First, follow the strongest links up to the second layer to identify 3 pooling groups the first-layer
neuron belongs to (left). Second, visualize each pooling group by identifying the Layer 1 neurons
that have the strongest links back from each Layer 2 neuron (right). In this example, the first pooling
group contains neurons marked with green, the second pooling group with red, and the third consist
of purple neurons. Colors correspond to rows in Figure 4 and both phases were performed for each
set (a)-(d).

15

