Randomness in Deconvolutional Networks for Visual Representation

Kun He1,2 , Jingbo Wang1 , Haochuan Li3 , Yao Shu1 , Mengxiao Zhang3 , Man Zhu1 , Liwei Wang3 , John E. Hopcroft2
1

School of Computer Science, Huazhong University of Science and Technology, Wuhan, China;
Department of Computer Science, Cornell University, Ithaca, USA;
3
Department of Machine Intelligence, Peking University, Beijing, China.
Correspondence to: Kun He: brooklet60@hust.edu.cn, Liwei Wang: wanglw@cis.pku.edu.cn.

arXiv:1704.00330v3 [cs.CV] 20 Feb 2018

2

Abstract
Toward a deeper understanding on the inner work
of deep neural networks, we investigate CNN
(convolutional neural network) using DCN (deconvolutional network) and randomization technique, and gain new insights for the intrinsic property of this network architecture. For the random
representations of an untrained CNN, we train
the corresponding DCN to reconstruct the input
images. Compared with the image inversion on
pre-trained CNN, our training converges faster
and the yielding network exhibits higher quality
for image reconstruction. It indicates there is rich
information encoded in the random features; the
pre-trained CNN may discard information irrelevant for classification and encode relevant features
in a way favorable for classification but harder for
reconstruction. We further explore the property of
the overall random CNN-DCN architecture. Surprisingly, images can be inverted with satisfactory
quality. Extensive empirical evidence as well as
theoretical analysis are provided.

1. Introduction
Since the first introduction in 1990’s (LeCun et al., 1989),
convolutional neural networks (CNNs) have demonstrated
impressive performance for various computer vision tasks,
and an essential element is to understand the deep representations for intermediate layers. A variety of visualization techniques have been developed in order to unveil the
feature representation and hence the inner mechanism of
CNNs (Zeiler & Fergus, 2014; Mahendran & Vedaldi, 2015;
Yosinski et al., 2015; Xu et al., 2015), however, our understanding of how these deep learning models operate remains
limited. In this paper we propose to apply randomization,
the assignment of random weights, on deconvolutional networks for the visual representation, and permit deeper understanding on the intrinsic property of the convolutional

architecture. Two techniques are closely related to our work,
deconvolution and randomization.
Deconvolutional networks (DCNs) are commonly used for
deep feature visualization. In the seminal work of Zeiler
et al. (2014), they propose to use a multi-layered deconvolutional network (Zeiler et al., 2011) to project the feature
activations back to the input pixel space, and show that the
features have many intuitively desirable properties such as
compositionality, increasing invariance and class discrimination for deeper layers. Dosovitskiy et al. (2016) design a
deconvolution variant which they call the up-convolutional
neural network to invert image representations learned from
a pre-trained CNN, and conclude that features in higher
layers preserve colors and rough contours of the images
and discard information irrelevant for the classification task
that the convolutional model is trained on. As there is no
back propagation, their reconstruction is much quicker than
the representation inverting method based on gradient descent (Mahendran & Vedaldi, 2015).
Randomization on neural networks can be tracked back to
the 1960’s where the bottom-most layer of shallow networks
consisted of random binary connections (Block, 1962). In
recent years, largely motivated by the fact that “randomization is computationally cheaper than optimization”, randomization has been resurfacing repeatedly in the machine
learning literature (Scardapane & Wang, 2017). For optimization problems such as regression or classification, this
technique is used to stochastically assign a subset of the
network weights to derive a simpler problem (Igelnik & Pao,
1995; Rahimi & Recht, 2009). Specifically, they compute
a weighted sum of the inputs after passing them through a
bank of arbitrary randomized nonlinearities, such that the
resulting optimization task is formulated as a linear leastsquares problem. Empirical comparisons as well as theoretical guarantees are provided for the approximation (Rahimi
& Recht, 2008; 2009; Arora et al., 2014). Other related
works include random kernel approximation (Rahimi &
Recht, 2007; Sinha & Duchi, 2016).

Randomness in Deconvolutional Networks for Visual Representation

Specifically on convolutional neural networks (CNNs), there
are a few lines of work considering randomization. Jarrett et
al. (2009) observe that, on a one-layer convolutional pooling
architecture, random weights perform only slightly worse
than pre-trained weights. Andrew et al. (2011) prove that
certain convolutional pooling architectures with random
weights are inherently frequency selective and translation
invariant, and argue that these properties underlie their performance. He et al. (2016) accomplish three popular visualization tasks, image inversion, texture synthesize and style
transfer, using random weight CNNs. Daniely et al. (2016)
extend the scope from fully-connected and convolutional
networks to a broad family of architectures by building a
general duality between neural networks and compositional
kernel Hilbert spaces and proving that random networks
induce representations which approximate the kernel space.
Motivated by the intuition that “random net may be theoretically easier to comprehend than the complicated welltrained net”, and that it may reveal the intrinsic property of
the network architecture, we use randomization to explore
the convolution followed by deconvolution architecture, and
provide theoretical analysis on the empirical observations.
Our goal is toward a deeper understanding on the inner
mechanism of deep convolutional networks.
Our main contributions are as follows:
First, we propose a random weight CNN subsequently connecting a trained deconvolutional network(DCN) to reconstruct images. By means of CNN-DCN architecture, either
fixing the random weights assigned to CNN model or utilising the pre-trained CNN model, we train the corresponding DCN to invert the input image. The DCN architecture
uses the inverted layer sequence of the CNN, as in (Dosovitskiy & Brox, 2016). Compared with the inversion on
pre-trained CNN, the random approach can train the corresponding DCN model more quickly and invert the image
with higher quality. The results explicitly reveal that random weight CNN can encode rich feature information of
the inputs while pre-trained CNN may discard feature information irrelevant for classification and encode relevant
feature in a way favourable for classification but harder for
image reconstruction.
Second, we present the overall random CNN-DCN architecture to further investigate the randomness in CNNs, i.e.
there is no training at all for inverting the inputs that pass
their information through a random weight convolutional
network. Surprisingly, the image is inverted with satisfactory quality. The geometric and photometric feature of the
inputs are well preserved. We argue that this is due to the intrinsic property of the CNN-DCN architecture. We provide
empirical evidence as well as theoretical analysis on the
reconstruction quality, and bound the error in terms of the
number of random nonlinearities, the network architecture

and the distribution of the random weights.

2. Preliminaries
2.1. Deconvolutional network architecture
For the network architecture, we consider two typical CNNs
for the deconvolution, VGG16 (Simonyan & Zisserman,
2015) and AlexNet (Krizhevsky et al., 2012). A convolutional layer is usually followed by a pooling layer, except for
the last convolutional layer, Conv5. For consistency, we will
explore the output after the convolutional layer but before
the pooling layer. In what follows, “feature representation”
or “image representation” denotes the feature vectors after
the linear convolutional operator and the nonlinear activation operator but before the pooling operator for dimension
reduction.
We build a CNN-DCN architecture on the layer of feature
representation to be studied. The convolution operator of a
deconvolutional layer in DCN is the same as the convolution
operator in CNN, and an upsampling operator is applied in
DCN to invert the corresponding pooling operator in CNN,
as designed in (Dosovitskiy & Brox, 2016). We will focus
on the representations of the convolutional layers, as Dosovitskiy et al. build DCNs for each layer of the pre-trained
AlexNet and find that the predicted image from the fully
connected layers becomes very vague. Figure 1 illustrates
an example of the VGG16 Conv5-DeConv5 architecture,
where Conv5 indicates the sequential layers from Conv1
to Conv5. For the activation operator, we apply the leaky
ReLU nonlinearity with slope 0.2, that is, r(x) = x if x ≥ 0
and otherwise r(x) = 0.2x. At the end of the DCN, a final
Crop layer is added to cut the output of DeConv1 to the
same shape as the original images.

Figure 1. The CNN-DCN architecture for Conv5 of VGG16.

We build deconvolutional networks on both VGG16 and
AlexNet, and most importantly, we focus on the random features of the CNN structure when training the corresponding
DCN. Then we do no training for deconvolution and explore
the properties of the purely random CNN-DCN architecture.

Randomness in Deconvolutional Networks for Visual Representation

2.2. Random distributions
For the random weights assigned to CNN or DCN, we try
several Gaussian distributions with zero mean and various
variance to see if they have different impact on the DCN
reconstruction. Subsequent comparison shows that a small
variance around 0.015 yields minimal inverting loss. We
also try several other types of random distributions, Uniform,
Logistic, Laplace, to study their impact.
• The Uniform distribution is in [-0.04, 0.04), such that
the interval equals [µ − 3δ, µ + 3δ] where µ = 0 and
δ = 0.015 are parameters for Gaussian distribution.
• The Logistic distribution is 0-mean and 0.015-scale of
decay. It resembles the normal distribution in shape
but has heavier tails.
• The Laplace distribution is with 0 mean and 2 ∗ λ2
variance (λ = 0.015), which puts more probability
density at 0 to encourage sparsity.

3. Random-CNN-trained-DCN
3.1. Training method for DCN
For each intermediate layer, using the feature vectors of all
training images, we train the corresponding DCN such that
the summation of L2 -norm loss between the inputs and the
outputs is minimized. Let Φ(xi , w) represent the output
image of the DCN, in which xi is the input of the ith image
and w is the weights of the DCN. We train the DCN to get
the desired weights w∗ that minimize the loss. Then for a
feature vector of a certain layer, the corresponding DCN can
predict an estimation of the expected pre-image, the average
of all natural images which would have produced the given
feature vector.
X
w∗ = arg min L = arg min
(Φ(xi , w) − xi )2 (1)
w

w

i

Specifically, we initialize the DCN by the “MSRA”
method (He et al., 2015) based on a modified Caffe (Jia
et al., 2014; Dosovitskiy & Brox, 2016). We use the training
set of ImageNet (Deng et al., 2009) and the Adam (Kingma
& Ba, 2015) optimizer with β1 = 0.9, β2 = 0.999 with
mini-batch size 32. The initial learning rate is set to 0.0001
and the learning rate gradually decays by the “multistep”
training. The weight decay is set to 0.0004 to avoid overfitting. The maximum number of iterations is set at 200,000
empirically.
3.2. Results for inverting the representation
Training. We observe similar results for the training loss in
different layers. Take the Conv2-DeConv2 architecture for
elaboration, the loss curves during the training process are
shown in Figure 2. Figure 2(a) compares VGG and AlexNet

on random as well as pre-trained weights. The training for
reconstruction converges much quicker on random CNN and
yields slightly lower loss, and this trend is more apparent
on VGG. It indicates that by pre-training for classification,
CNN encodes relevant features of the input image in a way
favorable for classification but harder for reconstruction.
Also, VGG yields much lower inverting loss as compared
with AlexNet. Figure 2(b) shows that random filters of
different Gaussian distributions on CNN affect the initial
training loss, but the loss eventually converges to the same
magnitude. Figure 2(c) shows that the four different random
distributions with appropriate parameters acquire similar
reconstruction loss.
Generalization. We take 5000 samples from the training set
and validation set respectively from ImageNet, and compare
their average reconstruction loss. The statistics is as shown
in Figure 3, where Convk represents a Convk-DeConvk
architecture. Figure 3(a) shows that the VGG architecture is
good in generalization for the reconstruction, and random
VGG yields much less loss than pre-trained VGG. For representations of deeper layers, the inverting loss increases
significantly for pre-trained VGG but grows slowly for random VGG. This means that in deeper layers, the pre-trained
VGG discards much more information that is not crucial for
classification, leading to a better classifier but a harder reconstruction task. Figure 3(b) compares VGG and AlexNet
on the Conv3-DeConv3 architecture. It shows that Alexnet
is also good in generalization for the reconstruction, and the
difference between the random and the pre-trained is very
small.
Reconstruction. Figure 4 shows reconstructions from various layers of random VGG and random AlexNet, denoted by
rwVGG and rwAlexNet respectively. 1 On both rwVGG and
rwAlexNet, the reconstruction quality decays for representations of deeper layers. The rwVGG structure yields more
accurate reconstruction, even on Conv5, which involves 26
convolution operations and 4 max pooling operations.
Figure 6 shows reconstructions from a cat example image for
various distributions of rwVGG Conv2-DeConv2. Except
for N (0, 1), the reconstruction quality is indistinguishable
by naked eyes. It shows that different random distributions
work well when we set the random weights relatively sparse.
In a nutshell, it is interesting that random CNN can speed up
the training process of the DCN on both VGG and AlexNet,
obtain higher quality reconstruction and generalize well for
other inputs. Regarding weights in the convolutional part
as a feature encoding of the original image, then the deconvolutional part can decode from the feature representations
encoded by various methods. The fact that the random en1
For input images, the cat is an example image from caffe, and
the other two are from the validation set of ImageNet.

Randomness in Deconvolutional Networks for Visual Representation

(a) VGG vs. Alexnet

(b) rwVGG

(c) rwVGG

Figure 2. Training loss for the Conv2-DeConv2 architecture.

(a) Random VGG vs. Pre-trained VGG

(b) VGG vs. Alexnet on Conv3

Figure 3. Comparison on the generalization error.
Original

Conv1

Conv2

Conv3

(a) rwVGG

Conv4

Conv5

Conv1

Conv2

Conv3

Conv4

Conv5

(b) rwAlexNet

Figure 4. (Zoom in for details.) Reconstructions for representations of different convolutional layers of rwVGG and rwAlexNet.

Figure 5. (Zoom in for details.) Reconstructions on rrVGG Conv1-DeConv1 networks.

Randomness in Deconvolutional Networks for Visual Representation

4. Random-CNN-random-DCN
In this section we mainly focus on VGG16, and explore the
reconstructions on purely random VGG CNN-DCN architecture (denoted by rrVGG for brevity). Surprisingly, we find
that even the overall CNN-DCN network is randomly connected, the images can be inverted with satisfactory quality.
In other words, the CNN randomly extracts the image features and passes them to the DCN, then in an unsupervised
manner the DCN can reconstruct the input image by random
feature extraction again. Our surprising results show that
the random-CNN-random-DCN architecture substantially
contributes to the geometric and photometric invariance
for the image inversion. This reminds us the immortal AI
Koan (Rahimi & Recht, 2009): Sussman said,“I am training
a randomly wired neural net to play tic-tac-toe,” “The net is
wired randomly as I do not want it to have any preconceptions of how to play.” Minsky said, “I will close my eyes so
the room will be empty.” And Sussman was enlightened.
In the following, we will systematically explore the reconstruction ability of the rrVGG architecture. For convenience analysis, we use the normal ReLU nonlinearity
(i.e. ReLU (x) = max(x, 0)), and the dimensions of the
convolutional layers of VGG are summarized in Table 1.
For evaluation, we use the Pearson correlation coefficient,
corr ∈ [−1, 1], between the output image and the original
image to approximately evaluate the reconstruction quality.
An extreme corr, either positive or negative, indicates a
high correlation with the geometric information. For statistics on the trend, we use the structural similarity (SSIM)
index (Wang et al., 2004), which is more accurate by considering the correlation and dependency of local spatially
close pixels, and more consistent to the perceptron of human
eyes. To remove the discrepancy on colors, we transform
the inputs and outputs in grey-scale, and in case of negative

4.1. Network width versus network depth
Taking the cat image as an example, we first study the reconstruction quality for different convolutional layers, as
shown in the first row of Figure 7. The weights are random in N (0, 0.1) and Convk indicates a Convk-DeConvk
architecture. The deeper the random representations are, the
coarser the reconstructed images are. Even though there is
no training at all, DCN can still perceive geometric positions
and contours for Conv1 to Conv3. We can still perceive a
very rough contour for the purely random Conv4-DeConv4
architecture, which is already 10 layers deep (Zoom in to
see the contours).
In the second row of Figure 7, we build a simplified inputConvk-DeConk-output architecture by connecting the data
layer directly to Convk followed by DeConvk, then to the
output layer. And we use the same N (0, 0.1) distribution
for the weights. The reconstruction quality is similar to
that of the first row, indicating that the dimension (width)
of the convolutional layer contributes significantly to the
reconstruction quality, while the depth of the CNN-DCN
network contributes only a little.
Layer
Data
Conv1
Conv2
Conv3
Conv4
Conv5

# Channels
3
64
128
256
512
512

Shape of feature maps
227 × 227
227 × 227
114 × 114
57 × 57
29 × 29
15 × 15

Dimension
154,587
3,297,856
1,663,488
831,744
430,592
115,200

Table 1. The dimension of each convolutional layer of VGG.
Conv1

Conv2

Conv3

Conv4

Conv5

Original

Figure 6. (Zoom in for details.) Reconstructions for representations of rwVGG Conv2 in various random distributions.

SSIM value, we invert the luminosity of the grayscale image
for calculation, so the value is in [0, 1].

Simplified

coding of CNN is easier to be decoded indicates that the
training for classification moves the image features of different categories into different manifolds that are moving
further apart. Also, it may discard information irrelevant
for the classification. The pre-trained CNN benefits the
classification but is adverse to the reconstruction.

Figure 7. (Zoom in for details.) Reconstruction images for the
rrVGG architecture, and the simplified rrVGG architecture. All
random weights are in N (0, 0.1) distribution.

As the reconstruction quality decays quickly for deep layers,
we argue that it may due to the shape reduction of the feature maps. As shown in Table 1, the shape of feature maps,
while going through convolutional layers, will be reduced
by 1/4 except for the input data layer. The convolutional
layer will project information of the previous layer to a 1/4
scale shape. So the representations encoded in feature maps
will be compressed randomly and it is hard for a random

Randomness in Deconvolutional Networks for Visual Representation

DCN to extract these feature representations. However in
the previous section, we see that the trained DCN can extract
these random deep representations easily and well reconstruct the images. It indicates that almost all information is
transformed to the next layer in the random CNN. Without
training on DCN, however, the reconstruction through low
dimension layers will certainly lose some information.
4.2. Results on number of random channels
We further explore the reconstruction quality on the number
of random channels using the rrVGG Conv1-DeConv1 architecture, which contains two convolutional operators, one
pooling operator, one upsampling operator and another two
convolutional operators. For simplicity, for each network
instance we use the same number of channels in all layers
except the output layer. We vary the number of random
channels from 4, 8 up to 2048, and for each number of channels, we generate 30 rrVGG Conv1-DeConv1 networks and
all random weights are in N (0, 0.1) distribution. For input
images we randomly pick 50 samples from the ImageNet
validation set.
To reduce occasionality on the reconstruction, we transform
the inputs and outputs in grey-scale and calculate the average SSIM value and Pearson correlation coefficient on each
network, then we do statistics (mean and standard deviation)
on the 30 average values. Figure 8 and Figure 9 show the
trends when the number of channels increases, (a) is for the
original rrVGG network and (b) is for the variant of rrVGG
network. The variant of rrVGG network is almost the same
as the original network except that the last convolutional
layer is replaced by an average layer, which calculates the
average over all the channels of the feature maps next to
the last layer. We see that the increasing number of random channels promotes the reconstruction quality. Similar
in spirit to the random forest method, different channels
randomly and independently extract some feature from the
previous layer, and they are complementary to each other.
With a plenty number of random channels we may encode
and transform all information to the next layer. The increasing trend and the convergence on the variant of the
rrVGG network are much more apparent. We will provide
theoretical analysis in the next section.
In Figure 5, we pick some input images, and show their
output images closest to the mean SSIM value for various
number of channels. The SSIM value is on the top of each
output image. As the color information is discarded for random reconstruction, we transform the images in grey-scale
to show their intrinsic similarity. The increasing number of
channels promotes the random reconstruction quality.

(a) original rrVGG

(b) variant rrVGG

Figure 8. Statistics on SSIM for rrVGG Conv1-DeConv1 architecture (mean, mean ± std). All weights are in N(0,0.1) distribution.

(a) original rrVGG

(b) variant rrVGG

Figure 9. Statistics on Pearson correlation for rrVGG Conv1DeConv1 architecture (mean, mean ± std). All weights are in
N(0,0.1) distribution.

4.3. Results on random kernel size
We expect that the reconstruction quality decays with larger
kernel size, as a large kernel size can not consider the local
visual feature of the input. In the extreme case when the
kernel size equals the image dimension, the convolution
operator actually combines all pixel values of the input to an
output pixel using random weights. To verify this assumption, we use the rrVGG Conv1 1 DeConv1 1 architecture,
which simply contains two convolutional operators. The
random weights are from the N (0, 0.1) distribution. For
each kernel size, we randomly generate 30 networks for the
reconstruction on 50 sample images as selected above.

Figure 10. SSIM for various kernel sizes on rrVGG Conv1 1.

5. Theories on Random Convolution
In this section, we provide theoretical analysis to explain
the surprising empirical results shown in the previous experiments. We will show that for a slight variant of the
random CNN architecture, when the number of channels in
each layer goes to infinity, the output image will converge

Randomness in Deconvolutional Networks for Visual Representation

to an image highly correlated to the input image. Note that
DCN is also a kind of CNN with up-sampling layers, so our
result can be directly applied to the CNN-DCN architecture.
We first introduce some notations and describe our variant
random CNN architecture.
Notations: We use A:,j to denote the j th column vector of
matrix A and kxk to denote the l2 -norm of vector x. Let L
be the number of layers in the neural network and X (i) ∈
RNi ×di be the feature maps in the ith layer, where Ni is
the number of channels and di is the dimension of a single
channel feature map (i.e. the width of the map times its
height). X = X (0) is the input image and f = X (L) is the
output image. w(i,j) , a row vector, is the j th convolutional
filter of the ith layer if it is a convolutional layer. We use
ReLU (x) = max(x, 0) as the activation function in the
following analysis.
Definition 5.1. Random CNN architecture This structure is different from the classic CNN structure in the following three points:
• Different filters in the same layer are i.i.d. random vectors and filters in different layers are independent. The
probability density function of each filter is isotropic.
(i,j) m

(i)

Let km = 12 E|w1
(i)

(i)

|

(i)

and Km =

(i)

(i) 2
k2m −(km
)
(i)

(km )2

. Sup-

(i)

pose that k1 , k2 , k4 all exist.
• The last layer is the arithmetic mean of the channels of
the previous layer, not the weighted combination.
(L−1)

, each layer of convolutional feature
• Except for X
maps are normalized by a factor of √1N , where Ni is
i
the number of channels of this layer.
5.1. Convergence
As from the previous experiments, we see that when the
number of channels increases, the quality of the output
image becomes better, so we are interested in what the
output image looks like when the number of channels in
each convolution layer goes to infinity. We prove that when
the number of channels goes to infinity, the output will
actually converge. Each pixel in the final convergence value
is a constant times the weighted norm of its receptive field.
Formally, we state our main results for the convergence
value of the random CNN as follows:
Theorem 5.1. (Convergence Value) Suppose all the pooling layers use l2 -norm pooling. When the number of filters
in each layer of a random CNN goes to infinity, the output f
corresponding to a fixed input will converge to a fixed image
f ∗ with probability 1, where f ∗ = kz∗ and k is a constant
only related to the CNN architecture
and the distribution of
qP
∗
2
random filters and zi =
l∈Ri n(l,i) kX:,l k , where Ri
∗
is the index set of the receptive field of zi and n(l,i) is the

number of routes from the lth pixel of a single channel of
the input image to the ith output pixel.
The proof of Theorem 5.1 is in the Appendix. Here for
the pooling layer, instead of average pooling, which calculates the arithmetic mean, we use l2 -norm pooling which
calculates the norm of the values in a patch. Intuitively, if
most pixels of the input image are similar to their adjacent
pixels, the above two pooling methods should have similar
outputs. We also show the result for average pooling in the
Appendix.
5.2. Variance
Now we consider the case of finite number of channels in
each layer. We mainly focus on the difference between
the real output and the convergence value as a function
of the number of channels. We prove that for our random
CNN architecture, as the number of channels increases, with
high probability, the angle between the real output and the
convergence value becomes smaller, which is in accordance
with the variant rrVGG experiment results shown in Section
4.2. To give an insight, we first state our result for a twolayer random CNN and then extend it to a multi-layer one.
Theorem 5.2. (Variance) For a two-layer random CNN
with N filters in the first convolutional layer, let Θ denote
the angle between the output f and the convergence
value
q
(0) 1
Nδ .

f ∗ , then with probability 1 − δ, sin Θ ≤

K1

Now we extend Theorem 5.2 to the multi-layer random
CNN setting:
Theorem 5.3. (Multilayer Variance) Suppose all the pooling layers use l2 -norm pooling. For a random CNN with
L layers and Ni filters in the ith layer, let Θ denote the
angle between the output f and the convergence value f ∗ ,
suppose that there is at most one route from an arbitrary
input pixel to an arbitrary output pixel for simplicity, then
with probability 1 − δ,
v
u
r
r
L−2
L−1 u
L−1 Y
t
+ (L − 2)
λi ,
sin Θ ≤
Nδ
N δ i=0
where λi =
PL−2
i=1

1
r

k(i) ((z∗(i) )2 )k2
1−
k(z∗(i) )2 k2

and

1
N

(L−2)

=

K1
1
L−1 ( NL−1

+

(i−1)

K2
Ni

).

Here, (i) (x) actually measures the local similarity of x.
The full definition of (i) (x) and the proof of Theorem 5.3
is in the Appendix.
5.3. Difference on the convergence value and the input
Finally, we focus on how well our random CNN architecture
can reconstruct the original input image. From Theorem

Randomness in Deconvolutional Networks for Visual Representation

5.2 and Theorem 5.3, we know that with high probability,
the angle between the output of a random CNN with finite
channels and the convergence value will be upper-bounded.
Therefore, to evaluate the performance of reconstruction, we
focus on the difference between the convergence value and
the input image. We will show that if the input is an image
whose pixels are similar to their adjacent pixels, then the
angle between the input image X and the convergence value
of the output image will be small. To show the essence
more clearly, we state our result for a two-layer random
CNN and leave the multi-layer one in the Appendix, which
needs more complicated technics but has the same insight
as the two-layer one.
Theorem 5.4. For a two-layer random CNN, suppose each
layer has a zero-padding scheme to keep the output dimension equal to the dimension of the original input. The kernel
size is r and stride is 1. The input image is X ∈ Rd0 ,
which has only one channel, whose entries are all positive. t = Xt − Xt means the difference between one pixel
Xt and the mean of the r-sized image patch whose center is Xt . Let Φ be the angle between the input image X
and the convergence
we have
P value of the output
P image,
1
2
cos Φ ≥ 1 − M

X
,
where
M
=
.
X
t
t
t
t t
The full proof of Theorem 5.4 is in the Appendix. Note
that when the kernel size r increases, t will become larger
as an image only has local similarity, so that the lower
bound of the cosine value becomes worse, which explains
the empirical results in Section 4.3.

6. Discussion and Conclusions
In recent years randomization has been attracting increasing
attention due to its competitive performance for optimization as compared to fully trained costly models, and the
potential for fast deep learning architecture selection. In
this work, we focus on understanding the inner representation of deep convolutional networks via randomization and
deconvolution. To our knowledge, this is the first attempt
to explore the randomness of convolutional networks by
inverting the representation back to the image space through
deconvolution and also the first attempt to explore the randomness of the overall CNN-DCN network structure in the
literature.
We first study the representation of random convolutional
networks by training a deconvolutional network, and shed
some lights on the inner mechanism of CNN. Extensive
empirical study shows that the representation of random
CNN performs better than that of the pre-trained CNN when
used for reconstruction. Our results indicate that the random
CNN can retain the photographically accurate information,
while the pre-trained CNN discards some irrelevant information for classification, making it harder for the image
reconstruction.

When we use random feature extractor for the overall CNNDCN architecture, with a plenty number of channels in each
layer, we can approximately reconstruct the input images.
We further explore the reconstruction ability of the purely
random CNN-DCN architecture from three perspectives:
the depth and width of the network, the number of channels and the size of the filter. As compared to the network
depth, the dimensions (width) of the convolutional layers
contribute significantly to the reconstruction quality. We investigate various number of channels for reconstruction and
find that the increasing number of random channels actually
promotes the reconstruction quality. For the third point, we
verify our assumption that the reconstruction quality will
decay with the increasement of the filter size.
In the end, we provide theoretical foundations to interpret
the empirical results on the random CNN-DCN architecture.
We prove that for a variant of the random CNN architecture
that calculates the average over the feature maps next to the
last layer, when the number of channels in each layer goes to
infinity, the output image will converge to an image highly
correlated to the input. We also bound the error between the
convergence value and the actual output when the number
of channels in each layer is finite.

Acknowledgments
This work was supported by National Natural Science Foundation of China (61602196, 61472147), US Army Research
Office (W911NF-14-1-0477).

A. Proof of Theories on Random Convolution
In the appendix, we prove that for a slight variant of the
random CNN architecture, when the number of filters in
each layer goes to infinity, the output will converge to a
fixed image f ∗ with probability 1 and it is proportional
to the square root of the weighted sum of the square of
the pixels in the corresponding receptive field; when the
numbers of filters in each layer are finite, then with high
probability, the angle between its output image and f ∗ is
bounded. For random convolutional neural networks with
zero-padding in each layer, we also give a lower bound for
the cosine value of the angle between the input image and
the convergence value of the output image.
For completeness, we repeat the notations and the definition
of random CNN architecture.
Notations: We use A:,j to denote the j th column vector
of matrix A and use Aij to denote its entry. Let xi be
the ith entry of vector x. Let L be the number of layers
in the neural network and X (i) ∈ RNi ×di be the feature
maps in the ith layer, where Ni is the number of channels
and di is the dimension of a single channel feature map

Randomness in Deconvolutional Networks for Visual Representation

(i.e. the width of the map times its height). X = X (0) is
the input image and f = X (L) is the output image. For
convenience, we also define convolutional feature maps to
be the feature maps after convolutional operation and define
pooled feature maps and up-sampled feature maps in the
same way. In the ith layer, let ri be the fixed kernel size
or the pool size (e.g. 3 × 3). If X (i+1) is convolutional
˜
feature maps, let Y (i) ∈ RNi ri ×di be the patched feature
(i)
˜
maps of X , where di is the number of patches and in fact
(i)
d˜i = di+1 and Y:,j is the receptive field of the j th pixel in
a single channel output image after the ith layer. To form
Y (i) , we first divide X (i) into patches and the mth patch
(i)
(i)
(i)
(i)
is X (i) , {X:,j | j ∈ Dm }, where Dm is an ordered
:,Dm

(i)
Dm,s

th

means the s corresponding element
Sd˜i
(i)
in
∈ [ri ]. We assume m=1
Dm = [di ], which
is satisfied by most widely used CNNs. By transforming
˜
(i)
(i)
X (i) into a column vector, we can obtain Y:,m . z(i) ∈ Rdi
set of indexes.
(i)
Dm , s

:,Dm

(i)

(i)

is a row vector defined by zj = kY:,j k, where k·k is the l2 norm operation. w(i,j) , a row vector, is the j th convolutional
filter of the ith layer. If X (i+1) is pooled feature maps, we
(i)
also divide X (i) into patches and Dm is the indexes of
th
(i+1)
the m patch. If X
is up-sampled feature maps we
(i)
have Dm = {m}, which has only one element. So we
can also define Y (i) and z(i) for pooling and up-sampling
layers. For the j th pixel of the output image in the last
layer, define its receptive filed on the input image in the first
layer as X:,Rj = {X:,m | m ∈ Rj }, where Rj is a set of
indexes. The activation function ReLU (x) = max(x, 0) is
the element-wise maximum operation between x and 0 and
(·)m is the element-wise power operation.

in each layer of a random CNN goes to infinity, the output f
corresponding to a fixed input will converge to a fixed image
f ∗ with probability 1, where f ∗ = kz∗ and k is a constant
only related to the CNN architecture
and the distribution
qP
2
of random filters and z∗i =
l∈Ri n(l,i) kX:,l k , where
n(l,i) is the number of routes from the lth input pixel to the
ith output pixel.
Here for the pooling layer, instead of average pooling, which
calculates the arithmatic mean, we use l2 -norm pooling
which calculates the norm of the values in a patch. We also
show the result for average pooling in Theorem A.7.
To prove the theorem, we first prove the following lemma.
Lemma A.2. Suppose w ∈ Rn , n ≥ 2 is a random
row vector and its probability density function is isotropic.
Y ∈ Rn×d is a constant matrix whose ith column vector is
denoted by yi . z ∈ Rd is a row vector and zi = kyi k. Let
g = max{wY, 0}. If km = 12 E|w1 |m exists, then Egm =
km zm and Egi gj = π1 k2 [(π − θij ) cos θij + sin θij ]zi zj ,
where θij is the angle between yi and yj .
Proof. Note that max{·, ·} and (·)m are both element wise
operations. The ith element of Egm is
(Egm )i = Emax{wyi , 0}m .
Since the probability density function of w is isotropic,
we can rotate yi to yi0 without affecting the value of
Emax{wyi , 0}m . Let yi0 = (kyi k, 0, ..., 0)T , we have
(Egm )i = Emax{kyi kw1 , 0}m
m
= zm
i Emax{w1 , 0}
1
= zm
E|w1m |
i
2
= k m zm
i .

Definition A.1. Random CNN architecture. This structure is different from the classic CNN in the following three
points:
• Different filters in the same layer are i.i.d. random vectors and filters in different layers are independent. The
probability density function of each filter is isotropic.
Let

(i)
km

pose

(i,j)
= 12 E|w1 |m and
(i) (i) (i)
k1 , k2 , k4 all exist.

(i)
Km

(i)

=

(i) 2
k2m −(km
)
(i)

(km )2

. Sup-

• The last layer is the arithmetic mean of the channels of
the previous layer, not the weighted combination.
• Except for X (L−1) , each layer of convolutional feature
maps are normalized by a factor of √1N , where Ni is
i
the number of channels of this layer.

Where the third equality uses the fact that the marginal
distribution of w1 is also isotropic. Similarly, we have:
Egi gj = Emax{wyi , 0}max{wyj , 0}.
We can also rotate yi and yj to yi0 and
yj0 .
Let yi0 = (kyi k, 0, 0, ..., 0)T and yj0
=
T
and suppose the
(kyj k cos θij , kyj k sin θij , 0, ..., 0)
marginal probability density function of (w1 , w2 ) is p(ρ)
which
pdoes not depend on φ since it is isotropic, where
ρ = w12 + w22 is the radial coordinate and φ is the angular
coordinate. We have:
Z ∞
Z π2
3
Egi gj = zi zj
p(ρ)ρ dρ
cos(θij − φ) cos φdφ.
0

A.1. Convergence
Theorem A.1. (Convergence Value) Suppose all the pooling layers use l2 -norm pooling. When the number of filters

Note that:
Z ∞
0

p(ρ)ρ3 dρ =

θij − π
2

1
1
2
Eρ2 = Ew12 = k2 ,
2π
π
π

Randomness in Deconvolutional Networks for Visual Representation

Z

π
2

cos(θij −φ) cos φdφ =
θij − π
2

1
((π−θij ) cos θij +sin θij ).
2

We obtain the second part of this lemma.
Now, we come to proof of Theorem A.1.

(i)

Proof. According to Lemma A.2, if X (i+1) is convolutional
feature maps, we can directly obtain:
1
Emax{w(i,1) Y (i) , 0}2
Ni+1

(i+1) 2

) =

E(X1,:

pooling or upsampling operation on the feature maps. The
only difference is that in a convolutional layer, instead of
calculating the inner product of a filter and the vector in
a sliding window, we simply calculate the l2 -norm q
of the

vector in the sliding window and then multiply it by k2 .
Note that z∗(0) can be directly obtained from the input image. Repeat this process layer by layer and we can obtain
z∗(L−2) .
According to Lemma A.2, we have:

(i)

k2
(z(i) )2 , 0 ≤ i ≤ L − 2
Ni+1

=

(L−1)

EX1,:

where we have fixed Y (i) and the expectation is taken over
random filters in the ith layer only. Since different channels
in X (i+1) are i.i.d. random variables, according to the strong
law of large numbers, we have:

(L−2) (L−2)

= E max{w(L−2,1) Y (L−2) , 0} = k1

z

.

Suppose that z(L−2) has converged to z∗(L−2) , and by DefiPNL−1 (L−1)
1
, we have:
nition A.1, f = NL−1
i=1 Xi,:

Ni+1

X

(i+1) 2 a.s.

(Xj,:

(i)

) −→ k2 (z(i) )2

when Ni+1 → ∞,

a.s.

(L−2) ∗(L−2)

f −→ k1

z

when Ni → ∞,

i ∈ [L − 1].

j=1

which implies that with probability 1,
Ni+1

lim

Ni+1 →∞

(z(i+1)
)2
m

=

X

lim

Ni+1 →∞

(i+1)

l∈Dm
(i)

= k2

X

(i+1) 2

(Xj,l

)

j=1

(i)

X

(zl )2 .

(i+1)

l∈Dm

Suppose that all Nj for 1 ≤ j ≤ i have gone to infinity and
z(i) has converged to z∗(i) , the above expression is the recur∗(i+1)
rence relation between zP
and z∗(i) in a convolutional
∗(i+1) 2
(i)
∗(i) 2
layer: (zm
) = k2
) .
(i+1) (z
l
l∈D

(L−2) QL−3 (i)
Let k
=
k1
and z∗
=
i=0 k2
QL−3 (i) −1 ∗(L−2)
∗
∗
( i=0 k2 ) z
, we have f
= kz . Note
that z∗ is obtained through a multi-layer sliding window
scheme similar to the CNN structure. It only depends on
the input image and the scheme. It is easy to verify that
z∗i is the square root of the weighted sum of the square
of input pixel values within the receptive field of the ith
output pixel, where the weight of an input image pixel is
the number of routes from it to the output pixel.

A.2. Variance

m

If X (i+1) is l2 -norm pooled feature maps, we have
(i+1)
(i)
kX:,l k = zl by defination. Therefore,
(z(i+1)
)2 =
m

f ∗ , then with probability 1 − δ, sin Θ ≤

(i)

X

Theorem A.3. (Variance) For a two-layer random CNN
with N filters in the first convolutional layer, let Θ denote
the angle between the output f and the convergence
value
q

(zl )2 .

(0) 1
Nδ .

K1

(i+1)

l∈Dm

(i)

If X (i+1) is up-sampled feature maps, a pixel Xjp will be
(i+1)

up-sampled to a r-sized block {Xjpq
(i+1)
Xjp1
(i+1)
D̃m

=

(i)
Xjp

| q ∈ [r]}, where

and all the other elements are zeros. Define
(i+1)

= {p | p1 ∈ Dm

}, we have:
X
(i)
(zl )2 .
(z(i+1)
)2 =
m

Proof. According to Theorem A.1, we have Ef = f ∗ =
(0)
k1 z∗ , where z∗ = z∗(0) . For a two-layer CNN, we can
directly obtain:

(i+1)

l∈D̃m

So far, we have obtained the recurrence relation in each
layer. In order to get z∗(i+1) given z∗(i) , we use the same
sliding window scheme on z∗(i) as that of the convolutional,

f=

N
N
1 X (1)
1 X
Xi,: =
max{w(0,i) Y (0) , 0},
N i=1
N i=1

Since different channels are i.i.d. random variables, we have

Randomness in Deconvolutional Networks for Visual Representation
(1)

(0)

1
ri+1
th

EXi,: = Ef = k1 z∗ . Then,

Define xm =

P

(i+1)

j∈Dm

xj , which is the average

(i)

value of the m patch. Let  (x)j = xj − xm , where m
1
(1)
(1)
(i+1)
EkXi,: − EXi,: k2
satisfies j ∈ Dm . We have:
N
1
(1)
(1)
X
X
= (EkXi,: k2 − kEXi,: k2 )
x2j =
(xm + (i) (x)j )2
N
(i+1)
(i+1)
d˜0
j∈Dm
j∈Dm
1 X
(0) 2
X
(0,i) (0)
2
∗ 2
= (
E max{w
Y:,j , 0} − (k1 ) kz k )
= ri+1 x2m +
(i) (x)2j ,
N j=1

Ekf − Ef k2 =

(i+1)

j∈Dm

d˜0
1 X
(0)
(0)
(0)
= (
k kY:,j k2 − (k1 )2 kz∗ k2 )
N j=1 2

(i+1)

|m ∈ [d˜i+1 ]} is a partition of [d˜i ] under our
Pd˜i+1 P
2
assumptions, we have kxk2 = m=1
(i+1) xj and
j∈Dm
P
˜
di+1 2
2
kφ(i) (x)k2 = ri+1
(k (i) )2 m=1
xm , which implies that
Since {Dm

1 (0)
(0)
(k − (k1 )2 )kz∗ k2
N 2
According to Markov inequality, we have:
=

1
(0)
(0)
(k − (k1 )2 )kz∗ k2 .
P r(kf − Ef k ≥  ) ≤
N 2 2
2

Let δ =
1 − δ:

(0)
1
N 2 (k2

2

≤ ri+1 (k (i) )2 kxk2 .

(0)

− (k1 )2 )kz∗ k2 , then with probability

kf − Ef k

≤ (0)
kEf k
k1 kz∗ k
v
r
u (0)
u k − (k (0) )2 1
(0) 1
2
1
t
=
= K1
.
(0) 2
Nδ
Nδ
(k )

sin Θ ≤

1

To extend the above two-layer result to a multi-layer one,
we first prove the following lemma. Note that in this lemma,
D(i) should be replaced by D̃(i) defined in the proof of
Theorem A.1 if X (i) is up-sampled feature maps.
˜

˜

Lemma A.4. Let φ(i) (·) : Rdi → Rdi+1 for 0 ≤ i ≤
L − 3 denote the recurrence relation of {(z∗(i) )2 } (i.e.
(z∗(i+1) )2 = φ(i) ((z∗(i) )2 )), then φ(i) (·) is a linear mapping. For simplicity, suppose that for any i ∈ [L − 3],
(i)
(i) T (i)
card(Dm ) = ri for any m ∈ [d˜i ]. And Dl
Dm =
∅ for any l, m ∈ [d˜i ] if l 6= m. Then we can obtain kφ(i) (x)k2 = ri+1 (k (i) )2 (kxk2 − k(i) (x)|2 ) ≤
(i)
ri+1 (k (i) )2 kxk2 , where k (i) = k2 for convolutional layers
and k (i) = 1 for pooling and up-sampling
layers and (i) (·)
P
1
(i)
is defined by  (x)j = xj − ri+1 l∈D(i+1) xl , where m
m

(i+1)

satisfies j ∈ Dm

kφ(i) (x)k2 = ri+1 (k (i) )2 (kxk2 − k(i) (x)|2 )

.

Proof. According to the definition of φ(i) (·) and Theorem
A.1, we have:
X
φ(i) (x)m = k (i)
xj .

Theorem A.5. (Multilayer Variance) Suppose all the pooling layers use l2 -norm pooling. For a random CNN with
L layers and Ni filters in the ith layer, let Θ denote the
angle between the output f and the convergence value f ∗ ,
suppose that there is at most one route from an arbitrary
input pixel to an arbitrary output pixel for simplicity, then
with probability 1 − δ,
v
u
r
r
L−2
L−1 u
L−1 Y
t
λi ,
sin Θ ≤
+ (L − 2)
Nδ
N δ i=0
where λi =

1
r
1−

1
N

and

k(i) ((z∗(i) )2 )k2
k(z∗(i) )2 k2
(L−2)

=

K1
1
L−1 ( NL−1

+

PL−2
i=1

(i−1)

K2
Ni

).

Proof. We will bound Θ recursively. Suppose that the angle
between (z(i) )2 and (z∗(i) )2 is θi . We have θ0 = 0. Let
PNi+1 (i+1,j)
(i+1)
1
g(i+1,j) = (Xj,: )2 and g(i+1) = Ni+1
.
j=1 g
If X (i+1) is convolutional feature maps, we have obtained
in the proof of Theorem A.1 that Eg(i+1) = Eg(i+1,j) =
(i)
k2 (z(i) )2 . Using similar method to the proof of Theorem A.3 and let αi+1 denote the angle between g(i+1) and
Eg(i+1) , we can derive that with probability 1 − δi+1 ,
s
sin αi+1 ≤

(i)

K2

1
.
Ni+1 δi+1

(i+1)

j∈Dm

di+1

It is easy to verify that for any c ∈ R and x, y ∈ R
we have φ(i) (cx) = cφ(i) (x) and φ(i) (x + y) = φ(i) (x) +
φ(i) (y). So φ(i) (·) is a linear mapping.

For a l2 -norm pooling layer or an up-sampling layer, we
have:
s
1
(i)
sin αi+1 = 0 ≤ K2
.
Ni+1 δi+1

Randomness in Deconvolutional Networks for Visual Representation

Let βi+1 denote the angle between g(i+1) and (z∗(i) )2 . We
have:
sin βi+1 ≤ sin(θi + αi+1 ) ≤ sin θi + sin αi+1 .
Note that there exists a constant γ such that sin βi+1 =
kγg(i+1) −(z∗(i) )2 k
. In fact, we can find the value of γ
k(z∗(i) )2 k
∗(i) 2
(g(i+1) )T
is (z kg)(i+1)
k2
(i)

k

((z
) )k
k(z∗(L−2) )2 k2

sin µ ≤

, where (·) means transpose. We use

kγ(z(i+1) )2 − (z∗(i+1) )2 k
≤
k(z∗(i+1) )2 k
=

1−

T

φ (·) to denote the recurrence relation of {(z∗(i) )2 }
(i.e. (z∗(i+1) )2 = φ(i) ((z∗(i) )2 )). Note that (z(i+1) )2 =
φ(i) (g(i+1) ) and φ(i) (·) is linear, using the result in Lemma
A.4, we can obtain:
sin θi+1

(L−1)

If we define d˜L−1 = 1 and D1
= [dL−1 ], then
L−2
dL−1
we
can
define
φ
(·)
:
R
→
R
as
φ(L−2) (x) =
P
∗(L−2) 2
L−2
k =φ
((z∗(L−2) )2 )
j∈[dL−1 ] xj . Note that kz
by definition. Then according to Lemma A.2, let λL−2 =
1
r
, we have
(L−2)
∗(L−2) 2 2

kγg(i+1) − (z∗(i) )2 k
k(z∗(i) )2 k2 − k(i) ((z∗(i) )2 )k2

λL−2 sin θL−2

Let v denote the angle between z(L−2) and f , we have obtained its bound inqTheorem A.3. With probability 1 − δL−1 ,

we have sin v ≤

above, define N by

(L−2)

K1
1
N

1
NL−1 δL−1 .
(L−2)

=

K1
1
L−1 ( NL−1

(i)
δN K2

kφ(i) (γg(i+1) − (z∗(i) )2 )k
kφ(i) ((z∗(i) )2 )k

≤p

p

With all the bounds
PL−2 K (i−1)
+ i=1 2Ni ) and
δN K

(L−2)

1
choose δi = (L−1)Ni for i ≤ L−2 and δL−1 = (L−1)N
L−1
for simplicity, we can obtain the bound of Θ: with probability 1 − δ,
v
u
r
r
L−2
L−1 u
L−1 Y
t
λi
sin Θ ≤
+ (L − 2)
Nδ
N δ i=0

= λi sin βi+1
≤ λi (sin θi + sin αi+1 ).
Where we defined λi =

1
r

k(i) ((z∗(i) )2 )k2
1−
k(z∗(i) )2 k2

for 0 ≤ i ≤

L − 3, which is usually slightly bigger than 1 if the input is
a natural image.
So far, we have derived the recurrence relation between
sin θi+1 and sin θi . So we can get the bound of θL−2 .
However, note that θL−2 is the angle between (z(L−2) )2
and (z∗(L−2) )2 instead of that between z(L−2) and z∗(L−2) .
We will denote the latter by µ which is what we really
need. We know that there exists a constant γ 0 such that
0
(L−2) 2
) −(z∗(L−2) )2 k
sin θL−2 = kγ (z k(z∗(L−2)
. Note that for any
)2 k
n

a, b ∈ R+ , according to Cauchy-Schwarz inequality, we
have:
n
n
X
X
2
2 2
n
(ai − bi ) ≥ (
|a2i − b2i |)2
i=1

i=1
n
X

=(

|ai − bi |(ai + bi ))2

i=1
n
X
≥ ( (ai − bi )2 )2 ,
i=1

which implies that ka − bk4 ≤ nka2 − b2 k2 . Then we can
bound µ:
√
k γ 0 (z(L−2) ) − (z∗(L−2) )k
sin µ ≤
kz∗(L−2) k
vq
u
u d˜ k(z∗(L−2) )2 k
L−2
t
≤
sin θL−2
kz∗(L−2) k2

A.3. Difference of the convergence value and the input
For this part, we will give a detailed proof for a two-layer
CNN and argue that the result can be directly extended to a
multi-layer one only with a few slight changes of definition.
Theorem A.6. For a two-layer random CNN, suppose that
each layer has a zero-padding scheme to keep the output
dimension equal to the dimension of the original input. The
kernel size is r and stride is 1. The input image is X ∈ Rd0 ,
whose entries are all positive. t = Xt − Xt means the
difference between one pixel Xt and the mean of the rsized image patch whose center is Xt . Let Φ be the angle
between the input image X and the convergence
value of
P
1
the output
image,
we
have
cos
Φ
≥
1
−

X
t t t , where
M
P
M = t Xt2 .
Proof. According to Theorem A.1, we know that f ∗ = kz∗ ,
where zt is the square root of the sum of the square of the
pixels in its corresponding receptive
field (i.e. the l2 -norm
qP
2
of the pixels), which means zt =
α∈Rt Xα and k is a
constant related to the network structure and distribution of
the random filters. With zero-padding on the input image,
we can calculate the angle between f ∗ and X:

P qP
2
t
α∈Rt Xα Xt
cos Φ = qP P
.
pP
2
2
X
X
t
α
t
α∈Rt
t
As each pixel contributes at most r times to the final output,
we have
sX X
√ √
Xα2 ≤ r M .
t

α∈Rt

Randomness in Deconvolutional Networks for Visual Representation

Also, using Cauchy-Schwarz Inequality and the fact that all
Xt are positive, we can obtain
sX
1 X
Xα2 ≥ √
Xα .
r
α∈Rt

α∈Rt

Now, we can bound the above cos Φ as follows:

P  1
P
√ Xt
t
α∈Rt Xα
r
cos Φ ≥
√ √ √
r M M
1 X
=
Xt Xt
M t
1 X
=
Xt (Xt − t )
M t
1 X
t Xt .
=1−
M t

The above theorem indicates that, if the input is an image
whose pixels are similar to their adjacent pixels, then the
the angle between the input image X and the convergence
value of the output image will be small.
We point out that the above theorem can be directly extended
to multi-layer convolutional neural networks. Suppose that
the neural network has multiple layers. According to Theorem A.1, f ∗ = kz∗ . Now, the pixels in the receptive fields
contribute unequally to the corresponding output. Note that
when the network is multi-layer, the receptive field is greatly
enlarged. We can similarly obtain:

P qP
2
t
α∈Rt n(α,t) Xα Xt
cos Φ = qP P
.
pP
2
2
n
X
X
(α,t)
α
t
α∈Rt
t t
Here, Rt is the index set of the receptive field of ft and
n(α,t) is the number of routes from Xα to ft . Suppose that
the receptive field of each ft has the same size and shape, Xt
is at a fixed relative position of the receptive field of ft and
n(α,t) only depends
on the relative position between Xα
P
and Xt . Let Xt =

n(α,t) Xα
α∈R n(α,t)

α∈Rt
P

be the weighted average

t

and t = Xt − Xt . By using the same technique above, we
can obtain that
1 X
cos Φ ≥ 1 −
t Xt .
M t
Note that although the bound is the same as the two-layer
convolutional neural network, as the receptive field is enlarged, t can be much larger, so that the above bound will
be worse.

We also give the convergence value for average pooling in
the next theorem.
Theorem A.7. (Convergence Value, average pooling)
Suppose all the pooling layers use average pooling. When
the number of filters in each layer of a random CNN goes
to infinity, the output f corresponding to a fixed input will
converge to a fixed image f ∗ with probability 1.

(i)

(i)

(i)

Proof. Define C (i) ∈ Rdi ×di by Cjk = (X:,j )T X:,k . If
X (i+1) is convolutional feature maps, according to Lemma
A.2, we have:
(i+1)

(i+1)

EX1,j X1,k
1
(i)
(i)
=
Emax{w(i,1) Y:,j , 0}max{w(i,1) Y:,k , 0}
Ni+1
1 (i)
(i) (i) (i)
=
k h(ϕjk )zj zk ,
Ni+1 2
(i)

(i)

(i)

where ϕjk is the angle between Y:,j and Y:,k and we defined h(x) = π1 [(π − x) cos x + sin x] for abbreviation. We
have fixed Y (i) and the expectation is taken over random filters in the ith layer only. Since different channels in X (i+1)
are i.i.d. random variables, according to the strong law of
large numbers, we have:
Ni+1
(i+1)

Cjk

=

X

(i+1)

Xl,j

(i+1) a.s.

(i)

(i)

(i) (i)

−→ k2 h(ϕjk )zj zk

Xl,k

l=1

when Ni+1 → ∞.
Note that:
v X
v X
u
u
(i) 2
(i)
(i)
u
zj = u
kX
k
=
Cll ,
:,l
t
t
(i)

(i)

l∈Dj

l∈Dj

(i)

(i)

cos ϕjk =

Pri

(i)

(Y:,j )T Y:,k
(i) (i)

s=1

=

zj zk

C

(i)
(i)

(i)

Dj,s Dk,s

(i) (i)

.

zj zk

Suppose that all Nj for 1 ≤ j ≤ i have gone to infinity
and C (i) has converged to C ∗(i) , the above expressions are
the recurrence relation between C ∗(i+1) and C ∗(i) for a
convolutional layer. If X (i+1) is average-pooled feature
maps, we have:
(i+1)

X:,j

=

1 X
(i)
X:,l .
ri
(i)
l∈Dj

Randomness in Deconvolutional Networks for Visual Representation

We have:
(i+1)

Cjk

(i+1) T

= (X:,j
1
= 2
ri

He, Kaiming, Zhang, Xiangyu, Ren, Shaoqing, and Sun,
Jian. Delving deep into rectifiers: Surpassing humanlevel performance on imagenet classification. In ICCV,
pp. 1026–1034, 2015.

(i+1)

) X:,k
X

(i)

(i)
(X:,l )T X:,m

He, Kun, Wang, Yan, and Hopcroft, John. A powerful generative model using random weights for the deep image
representation. In NIPS, pp. 631–639, 2016.

(i)

(i)

l∈Dj ,m∈Dk

=

1
ri2

(i)

X

Clm ,
(i)

(i)

l∈Dj ,m∈Dk

which is the recurrence relation for an average pooling layer.
(i)

For an up-sampling layer, a pixel Xjk will be up-sampled
(i+1)

(i+1)

(i)

to a block {Xjkm | m ∈ [r]}, where Xjk1 = Xjk and
all the other elements are zeros. We have:
(
(i)
Cjk for l = m = 1,
(i+1)
Cjl km =
0
otherwise.
∗(0)

Note that we can directly calculate C
according to the
input image. So we can recursively obtain C ∗(L−2) and thus
z∗(L−2) .
According to Lemma A.2, we have:
(L−1)
EX1,:

= E max{w

(L−2,1)

Y

(L−2)

, 0} = k1 z

(L−2)

.

Suppose that z(L−2) has converged to z∗(L−2) , and by DefiPNL−1 (L−1)
1
nition A.1, f = NL−1
, we have:
i=1 Xi,:
a.s.

f −→ k1 z∗(L−2)

when Ni → ∞,

i ∈ [L − 1].

We can obtain the convergence value f ∗ through the above
process.

References
Arora, Sanjeev, Bhaskara, Aditya, Ge, Rong, and Ma,
Tengyu. Provable bounds for learning some deep representations. In ICML, pp. 584–592, 2014.
Block, H.D. Perceptron: A model for brain functioning.
Review of Modern Physics, 34:123–135, 1962.
Daniely, Amit, Frostig, Roy, and Singer, Yoram. Toward
deeper understanding of neural networks: The power of
initialization and a dual view on expressivity. In NIPS,
pp. 2253–2261, 2016.
Deng, Jia, Dong, Wei, Socher, Richard, Li, Li-Jia, Li, Kai,
and Li, Fei-Fei. Imagenet: A large-scale hierarchical
image database. In CVPR, pp. 248–255, 2009.
Dosovitskiy, Alexey and Brox, Thomas. Inverting visual
representations with convolutional networks. In CVPR,
pp. 4829–4837, 2016.

Igelnik, Boris and Pao, Yoh-Han. Stochastic choice of
basis functions in adaptive function approximation and
the functional-link net. IEEE Trans. Neural Networks, 6
(6):1320–1329, 1995.
Jarrett, Kevin, Kavukcuoglu, Koray, Ranzato, Marc’Aurelio,
and LeCun, Yann. What is the best multi-stage architecture for object recognition? In ICCV, pp. 2146–2153,
2009.
Jia, Yangqing, Shelhamer, Evan, Donahue, Jeff, Karayev,
Sergey, Long, Jonathan, Girshick, Ross, Guadarrama,
Sergio, and Darrell, Trevor. Caffe: Convolutional architecture for fast feature embedding. In ACM MM, pp.
675–678, 2014.
Kingma, Diederik and Ba, Jimmy. Adam: A method for
stochastic optimization. In ICLR, 2015.
Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey E.
Imagenet classification with deep convolutional neural
networks. In NIPS, pp. 1097–1105, 2012.
LeCun, Yann, Boser, Bernhard E., Denker, John S., Henderson, Donnie, Howard, Richard E., Hubbard, Wayne E.,
and Jackel, Lawrence D. Backpropagation applied to
handwritten zip code recognition. Neural Computation, 1
(4):541–551, 1989.
Mahendran, Aravindh and Vedaldi, Andrea. Understanding
deep image representations by inverting them. In CVPR,
pp. 5188–5196, 2015.
Rahimi, Ali and Recht, Benjamin. Random features for
large-scale kernel machines. In NIPS, pp. 1177–1184,
2007.
Rahimi, Ali and Recht, Benjamin. Uniform approximation
of functions with random bases. In Allerton Conference
on Communication, Control, and Computing, pp. 555–
561, 2008.
Rahimi, Ali and Recht, Benjamin. Weighted sums of random kitchen sinks: Replacing minimization with randomization in learning. In NIPS, pp. 1313–1320, 2009.
Saxe, Andrew, Koh, Pang W, Chen, Zhenghao, Bhand, Maneesh, Suresh, Bipin, and Ng, Y. On random weights and
unsupervised feature learning. In ICML, pp. 1089–1096,
2011.

Randomness in Deconvolutional Networks for Visual Representation

Scardapane, Simone and Wang, Dianhui. Randomness in
neural networks: an overview. WIREs: Data Mining and
Knowledge Discovery, 7(2), 2017.
Simonyan, Karen and Zisserman, Andrew. Very deep convolutional networks for large-scale image recognition. In
ICLR, 2015.
Sinha, Aman and Duchi, John C. Learning kernels with
random features. In NIPS, pp. 1298–1306, 2016.
Wang, Zhou, Bovik, Alan C., Sheikh, Hamid R., and Simoncelli, Eero P. Image quality assessment: from error
visibility to structural similarity. IEEE Trans. Image Processing, 13(4):600–612, 2004.
Xu, Kelvin, Ba, Jimmy, Kiros, Ryan, Cho, Kyunghyun,
Courville, Aaron, Salakhudinov, Ruslan, Zemel, Rich,
and Bengio, Yoshua. Show, attend and tell: Neural image
caption generation with visual attention. In ICML, pp.
2048–2057, 2015.
Yosinski, Jason, Clune, Jeff, Nguyen, Anh, Fuchs, Thomas,
and Lipson, Hod. Understanding neural networks through
deep visualization. arXiv preprint arXiv:1506.06579,
2015.
Zeiler, Matthew D and Fergus, Rob. Visualizing and understanding convolutional networks. In ECCV, pp. 818–833,
2014.
Zeiler, Matthew D., Taylor, Graham W., and Fergus, Rob.
Adaptive deconvolutional networks for mid and high level
feature learning. In ICCV, pp. 2018–2025, 2011.

