Recursive Autoconvolution for Unsupervised
Learning of Convolutional Neural Networks
Boris Knyazev∗† , Erhardt Barth† and Thomas Martinetz†
∗

arXiv:1606.00611v2 [cs.CV] 26 Mar 2017

Bauman Moscow State Technical University
Email: bknyazev@bmstu.ru, borknyaz@gmail.com
†
Institut für Neuro- und Bioinformatik, University of Lübeck
Email: {barth, martinetz}@inb.uni-luebeck.de

Abstract—In visual recognition tasks, such as image classification, unsupervised learning exploits cheap unlabeled data and
can help to solve these tasks more efficiently. We show that the
recursive autoconvolution operator, adopted from physics, boosts
existing unsupervised methods by learning more discriminative
filters. We take well established convolutional neural networks
and train their filters layer-wise. In addition, based on previous
works we design a network which extracts more than 600k
features per sample, but with the total number of trainable
parameters greatly reduced by introducing shared filters in
higher layers. We evaluate our networks on the MNIST, CIFAR10, CIFAR-100 and STL-10 image classification benchmarks and
report several state of the art results among other unsupervised
methods.

I. I NTRODUCTION

n=0

n=1

n=0

n=1

n=2

n=0

n=1

n=2

n=0

n=1

(a)

(c)

n=2

(b)
n=2

(d)

Fig. 1: Recursive autoconvolution (Eq. 2) of orders n = 0, 1, 2 applied to
samples: MNIST (a); CIFAR-10 (b); STL-10 (c,d). Note how the patterns
obtained for n = 1, 2 (highlighted in green frames) are similar to the low and
mid-level features of CNNs (e.g., visualized in Fig. 2 in [15]). The novelty of
this work is that we learn filters from these patterns.

Large-scale visual tasks can now be solved with big deep
neural networks, if thousands of labeled samples are available steadily approaching performance levels of supervised models,
and if training time is not an issue. Efficient GPU implemen- including convolutional neural networks (CNNs) trained by
tations of standard computational blocks make training and backpropagation on thousands of labeled samples [11, 12, 13].
This way, we provide further evidence that unsupervised learntesting feasible.
A major drawback of supervised neural networks is that they ing is promising for building efficient visual representations.
The main contribution of this work is adaptation of the
heavily rely on labeled data. It is true that in real applications
recursive
autoconvolution operator [14] for convolutional
it does not really matter which methods are used to achieve
architectures.
Concretely, we demonstrate that this operator
the desired outcome. But in some cases, labeling can be an
can
be
used
together
with existing clustering methods (e.g., kexpensive process. However, visual data are full of abstract
means)
or
other
learning
methods (e.g., independent component
features unrelated to object classes. Unsupervised learning
analysis
(ICA))
to
train
filters
that resemble the ones learned by
exploits abundant amounts of these cheap unlabeled data and
CNNs
and,
consequently,
are
more
discriminative (Fig. 1, 2 and
can help to solve the same tasks more efficiently. In general,
Sections
III,
IV-A).
Secondly,
we
substantially
reduce the total
unsupervised learning is important for moving towards artificial
number
of
learned
filters
in
higher
layers
of
some
networks
intelligence [1].
without
loss
of
classification
accuracy
(Section
IV-A3),
which
In this work, we learn a visual representation model for
allows
us
to
train
larger
models
(such
as
our
AutoCNN-L32
image classification, which is particularly effective (in terms of
accuracy) when the number of labels is relatively small. As a with over 600k features in the output). Finally, we report
result, our model can potentially be successfully applied to other several state of the art results among unsupervised methods
tasks (e.g., biomedical data analysis or anomaly detection), in while keeping computational cost relatively low (Section V-D).
which label information is often scarce or expensive.
II. R ELATED WORK
We are inspired by the previous works in which network
Unsupervised learning is used quite often as an additional
filters (or weights) are learned layer-wise without label information [2, 3, 4, 5, 6, 7, 8]. In accordance with these works, regularizer in the form of weights initialization [16] or
we thoroughly validate our models and confirm its advantage reconstruction cost [13, 17], or as an independent visual model
in the tasks with only few labeled training samples, such as trained on still images [18, 10] or image sequences [19].
STL-10 [3] and reduced variants of MNIST [9] and CIFARHowever, to a larger extent, our method is related to another
10 [10]. In addition, on full variants of these datasets and series of works [2, 3, 4, 5, 20, 8] and, in particular, [6, 7],
on CIFAR-100 we demonstrate that unsupervised learning is in which filters (or some basis) are learned layer-wise and,

contrary to the methods above, neither backpropagation nor
fine tuning is used.
In these works, learning filters with clustering methods, such
as k-means, is a standard approach [3, 20, 8, 7, 6]. For this
reason, and to make comparison of our results easier, k-means
is also adopted in this work as a default method. Moreover,
clustering methods can learn overcomplete dictionaries without
additional modifications, such as done for ICA in [5]. Nevertheless, since ICA [21] is also a common practice to learn
filters, we conduct a couple of simple experiments with this
method, as well as with principal component analysis (PCA),
to probe our novel idea more thoroughly. In contrast to various
popular coding schemes [2, 3, 4, 8], our forward pass is built
upon a well established supervised method - a convolutional
neural network [9].
Recently, convolutional networks were successfully trained
layer-wise in an unsupervised way [6, 7]. In their works, as well
as in our work, the forward pass is mostly kept standard, while
methods to learn stronger (in terms of classification) filters
are developed. For instance, in [6], k-means is enhanced by
introducing convolutional clustering. Convolutional extension
of clustering and coding methods is one of the ways to
reduce redundancy in filters and improve classification, e.g.,
convolutional sparse coding [22]. In this work, we suggest
another concept of making filters more powerful, namely, by
recursive autoconvolution applied to image patches before
unsupervised learning.
Autoconvolution (or self-convolution) and its properties seem
to have been first analyzed in physics (in spectroscopy [23])
and later in function optimization [24] as the problem of
deautoconvolution arose. This operator also appeared in visual
tasks to extract invariant patterns [25]. But, to the best of our
knowledge, its recursive version, used as a pillar in this work,
was first suggested in [14] for parametric description of images
and temporal sequences. By contrast, we use this operator to
learn convolution kernels in a multilayer CNN, which we refer
to as an AutoCNN.

n=0

n=1

n=2

n=3

(a)
(b)
(c)
(d)
Fig. 2: Examples of filters D(1) of size 13 × 13 pixels learned using k-means
on the MNIST (a,b) and CIFAR-10 (c,d) patches with (b,d) and without (a,c)
whitening for recursive autoconvolution orders n = 0, 1, 2, 3. In each case,
16 filters are learned. Note that filters are more sparse for higher n. Similar
filters are learned in the first layer of our AutoCNN networks.

lost and the inverse operation becomes ill-posed [24]. In this
work, we do not address this problem.
To extract patterns from X, it is necessary to make sure that
mean(X) = 0 and std(X) > 0 before computing (1). Also,
to compute linear autoconvolution, X is first zero-padded, i.e.
for one dimensional case X have to be padded with zeros to
length (2s − 1), where s - length of X before padding.
A. Recursive autoconvolution
We adopt the recursive autoconvolution (RA) operator,
proposed earlier in [14], which is an extension of (1):
Xn = Xn−1 ∗ Xn−1 = F −1 (F(Xn−1 )2 ),

(2)

where n = [0, nmax ] - an index of the recursive iteration (or
autoconvolution order). If n = 0, X0 equals the input, i.e. a
raw image patch. For n = 1 Eq. 2 becomes equal Eq. 1. In
our work, we limit nmax = 3 as higher orders do not lead to
better classification results.
In [14], image patterns extracted using this operator were
used for parametric description of images. The novelty of this
work is that we use extracted patterns as convolution kernels,
i.e. filters, because we noticed that applying Eq. 2 with n ≥ 1
to image patches provides sparse wavelet-like patterns (Fig. 1,
2), which are usually learned by a CNN in the first layer (see
[12], Fig. 3 or [19], Fig. 6) or by other unsupervised learning
methods, e.g., ICA (see [5], Fig. 1), or sparse coding (see [2],
Fig. 2). In addition, these patterns are similar to more complex
mid-level features of CNNs (e.g., visualized in Fig. 2 in [15]).

III. AUTOCONVOLUTION
Our filter learning procedure, discussed further in Section
IV-A, is largely based on autoconvolution and its recursive
extension. In this and the next sections, we formulate the basics
of these operators in general terms.
IV. AUTO CNN ARCHITECTURE
We first describe the routine for processing arbitrary discrete
In this section, we describe the architecture of a multilayer
data based on autoconvolution. It is convenient to consider
convolutional neural network (AutoCNN), which we train
autoconvolution in the frequency domain. According to the
in an unsupervised way and then use as a feature extractor
convolution theorem, for N -dimensional discrete signals X
(Fig. 3). It is based on classical CNNs [9], however, there
and Y, such as images (N = 2): F(X ∗ Y) = kF(X) ◦ F(Y),
is no backward pass and neither filters nor connections
where F - the N -dimensional forward discrete Fourier transare trained using labels. In addition, based on [6, 7] we
form (DFT), ◦ - point-wise matrix product, k - a normalizing
design two networks (AutoCNN-S32 and AutoCNN-L32) with
coefficient (which will be ignored further, since we apply
features randomly split into several (32) groups (see Section
normalization afterwards). Hence, autoconvolution is defined
IV-A2), which is not typical for CNNs, but important in our
as
work. Since we have only forward pass, we are not limited
−1
2
X ∗ X = F (F(X) ),
(1)
by differentiable functions and canpadd nonlinearities such
where F −1 - the N -dimensional inverse DFT, ∗ - convolution. as Rootsift normalization (sign(x) |x|/kxk1 ) [26], which
Because of squared frequencies, some phase information is improves classification significantly for some datasets. Batch

Random crops: 5x5x3
Recursive autoconv
k-means

Random crops: 5x5x256
Recursive autoconv
PCA

Random crops: 5x5x32
Recursive autoconv
PCA

ReLU
MaxPool
LCN
Grouping

Next, according to our statistics of extracted patches, presented in Fig. 4(a), autoconvolution order n is inversely proporShared filters
tional to the joint spatial σxy and frequency
√ σuv resolution, i.e.
n ∼ 1/(σxy σuv ), where σxy = σx σy = D1 D2 and D1 , D2
SVM
- are eigenvalues
of the weighted covariance matrix of X in
the spatial domain; analogously for σuv . Therefore, to cover a
wider range of spatio-frequency properties and to learn a more
diverse set of filters, patches extracted with several orders are
INPUT from X :
32x32x3
combined into one global set. That is, we take results of several
Filters D :
Feature maps X : Filters D : Feat. maps X : Filters D :
Feat. maps X :
5x5x3x1024 16x16x32x32
5x5x32x256 8x8x256x32
5x5x256x1024 4x4x1024x32
orders (e.g., in case n = [0, 3] we have 4 patches instead of 1)
multidict
Fig. 3: Overview of our best architecture for CIFAR-10 (AutoCNN-L32). Filters and put them into one global set of autoconvolutional patches.
D(1) , D(2) and D(3) are learned with unsupervised Abs
learning and recursive
Note that in case n = 0, we extract more patches to make the
Abs
ZCA-whiten
(1) , X(2) are
Rootsift
Rootsift
MaxPool(2)
autoconvolution
(RA). Features XRandom
pooledMaxPool(2)
and concatenated
with
Batch norm
Batch norm
total numberRootsift
of input data points for k-means about the same
Batch
norm
LCN
Conv1 a large feature vector
grouping (672k Conv2
X(3) to form
values). Steps thatConv3
are usually
Randomized PCA
as
for
combinations
of orders. In this global set, all patches
missing in CNNs are in bold.
Stand
(a) AutoCNN-L2
are first scaled to have values in the range [0,1], then they
1x1500 1x50 1x75
1x1500
are ZCA-whitened as in [3, 6, 7, 10]. Even though, with RA
MaxPool(4)
standardization 1 (or ’batch norm’) is employed for all networks we can extract Gabor-like patterns without whitening (see Fig.
before convolutions - an entire batch is treated as a vector. In 2(a),(c)), such preprocessing is essential for k-means to learn
some cases (see details in Table VI) we followRecursive
[7] and apply less correlated filters. For this whitened set k-means produces
autoconvolution +
Local Contrast Normalization (LCN) between k-means
layers, but we a set of Kl data points (a dictionary) D(l) ∈ Rsl ×sl ×dl ×Kl .
do it after pooling for speedup.
These data points are first l2 -normalized and then used as
The architectures of all networks used in this work are convolution kernels (filters) for layer l.
summarized in Table VII.
2) Grouping of feature maps: Filters of the second and
following
layers can be trained either in the same way as of the
SVM
SVM SVM
A. Training
the network
first layer or using a slightly modified procedure borrowed from
Training
our network is analogous to previous works on layer- [6, 7]. Concretely, features X(l) , l > 1 can be split (randomly
fusion+classification
wise (b)
learning
[6, 7]. We start by learning filters D(1) for layer in this work) into G groups. It is useful in practice, because
SVM Committee
l
l = 1 using training (unlabeled) images X(0) , then process filters of layer l + 1 will have smaller depth d
l+1 = Kl /Gl
these images with the learned filters D(1) followed by rectifi- (instead of d
l+1 = Kl as in typical CNNs) and smaller overall
cation, pooling, normalization and further steps (depending on dimensionality, so that it is easier to learn such filters with kthe architecture) that yield the first layer feature maps X(1) . means. At the same time, the number of features becomes larger
Using these features, we train filters D(2) for layer 2, and then by factor G , and more features usually improve classification
l
process them X(1) with the learned filters D(2) , and so forth. in case of unsupervised learning.
Features concatenated from all layers (multidictionary features)
3) Shared filters for Gl > 1: The disadvantage of such
are used for classification with support vector machines (SVM). splitting is that filters have to be learned for each group indepenTraining filters D(1) ∈ Rs1 ×s1 ×d1 ×K1 for layer 1, that is dently, i.e. it is necessary to run k-means G times. For instance,
l
K1 filters of size s1 × s1 with d1 color channels, is performed for the second layer we have D(2) ∈ Rs2 ×s2 ×d2 ×K2 ×G1 , i.e.
in three steps: (1) random patches X(1) ∈ Rs1 ×s1 ×d1 are the total number of filters equals K × G . Our contribution
2
1
extracted from all training (unlabeled) samples; (2) recursive is that instead of treating groups independently, we learn
autoconvolution (Eq. 2) is applied to them; (3) one of the unsu- filters for all groups taken together without negative effect
pervised learning methods is applied to the autoconvolutional on classification accuracy (see results in Table Ib). In this
patches. Filters for the following layers are trained according case, patches from all G feature map groups are concatenated
1
to this procedure as well. Unless otherwise stated, k-means before clustering and k-means is run only once. Thus, the total
with Euclidean distance is employed for filter learning.
number of filters in layer 2 becomes equal K2 , i.e. the same
1) Learning with recursive autoconvolution (RA): Learning filters are shared between all G groups. This trick enables us
1
filters with RA is a novel idea, so we explain some steps of its to learn our large AutoCNN-L32 with G = 32 in a feasible
1
efficient application to our tasks. One of the issues with RA is time.
that the spatial size of image patches X is doubled after each
V. E XPERIMENTS
iteration n due to zero-padding (see Eq. 1 and 2). To learn
filters from patches, we need all the patches to have some fixed A. Experimental setup
size, so we simply take the central part of the result or resize
We evaluate our method on four image classification bench(subsample) it to its original size after each iteration (Fig. 1, marks: MNIST [9], CIFAR-10, CIFAR-100 [10] and STL-10
where the second option is picked). We randomly choose one [3]. To demonstrate that unsupervised learning is particularly
of these options to make the set of patches richer.
effective for datasets with few training samples, such as STLBatch norm
Conv
RA

Abs
Rootsift
Batch norm MaxPool
LCN
Conv
RA

OUTPUT:
672k features
Abs
MaxPool

Rootsift
Batch norm
Conv
RA

(0)

(1)

(1)

(2)

(2)

(3)

(3)

F2
MaxPool(2)
1x4*4*256*32
F1
MaxPool(4)
1x16384

1 Along this work, vector x is considered standardized if its mean(x) = 0
and std(x) = 1.

10, which has only 100 training images per class, we test our
method on reduced versions of MNIST and CIFAR-10, namely

200

100

0
0.006 0.1

0.2 0.3
σxy σuv

0.4

no RA (n = 0)
RA (n = 1)
RA (n = 2)
RA (n = [0, 3])

0.5

98.8
98.6
98.4
98.2
5

7
9 11 13
filter size (s1 ), px

15

(d) STL-10 (1k/8k, 96x96x3)

68
test accuracy, %

cross-validation accuracy, %

frequency

300

cross-validation accuracy, %

(b) MNIST (10k/10k, 28x28) (c) CIFAR-10 (10k/10k, 32x32x3)

(a) Distribution on CIFAR-10

66
64

58

56

54

62
5

7

9 11 13 15
filter size (s1 ), px

17

9

13 17 21 25
filter size (s1 ), px

29

no RA (K1 = 96; n = 0)
RA (K1 = 96; n = [1, 3])
no RA (K1 = 192; n = 0)

no RA (K1 = 96; n = 0)

RA (K1 = 96; n = [0, 3])

no RA (K1 = 256; n = 0)

RA (K1 = 256; n = [0, 3])

RA (K1 = 192; n = [1, 3])

no RA (K1 = 512; n = 0)

RA (K1 = 512; n = [0, 3])

Fig. 4: (a) Recursive autoconvolution (RA) makes filters more localized and sparse as shown by distributions of the joint spatial and frequency resolution
σxy σuv for the first layer filters learned on CIFAR-10 with k-means with different orders n. (b-d) For our simple single layer network (AutoCNN-S1) RA
improves results for a wide range of filter sizes s1 and number of filters (K1 ) on three datasets: MNIST (b), CIFAR-10 (c), STL-10 (d). Not that in some
cases results with RA are better than without RA even if the latter has two times more filters. The number of training/test samples and input image sizes in the
datasets are indicated in parentheses for reference.

MNIST (100), MNIST (300) and CIFAR-10 (400) with just In this experiment, in case of MNIST and CIFAR-10, 10100, 300 and 400 training images per class respectively. We fold cross-validation is performed with 10k training and 10k
follow the same experimental protocol as in previous works, test samples drawn from the original training sets. For STLe.g., [6, 18]: 10 random subsets (folds) from the training set 10 we use the original predefined folds. We observe that
are drawn, while the test set remains fixed. For STL-10 these recursive autoconvolution consistently improves classification.
folds are predefined. Unless otherwise stated, we report average It is evident especially for larger filters, because RA makes
classification accuracies (or errors on MNIST) in percent; on filters more localized and sparse (Fig. 4(a)). This experiment
reduced datasets, these results are averaged over 10 runs. In also shows that the results generalize well across different
Tables Ia-V better results are indicated in bold. Images of all datasets and preprocessing strategies.
datasets, except for MNIST, are ZCA-whitened as in most
We also experimented with each of the orders in Eq. 2
previous works (see details in Table VI).
independently (results not shown) to determine filters of which
While in previous works all labeled training samples are orders contribute the most during classification, but we found
typically used as unlabeled data during unsupervised learning, that, generally, combinations of orders work best.
we found that it is enough to use at most 10k-20k samples
to learn filters in all experiments, including STL-10, which TABLE I: (a) Comparison of learning algorithms on CIFAR-10(400) using
AutoCNN-S1-256. (b) Shared vs independent filters on CIFAR-10(400) for a
contains 100k unlabeled samples.
two layer model AutoCNN-S32 + RA (N - number of trainable parameters, † :
We train models with 1-3 layers according to the proposed time for learning filters with k-means for 32 groups measured on Intel Xeon
architecture (Section IV). The details of all models are CPU E5-2620v3.).
presented in Table VII. Network parameters (number of layers
(a)
(b)
and filters, pooling size and stride, etc.) are chosen so that to
Learning method Raw RA
Filters Avg acc N Train t †
make them more consistent with previous works [18, 27, 7, 6]
k-means
64.0 65.6
Independ. 72.0 432.5k 470s
PCA (s1 ≥ 11) 60.7 59.8
and within this work.
Shared
71.9 43.6k 20s
ICA (s1 ≥ 11)
62.8 64.5
In case of a multilayer network, we use multidictionary
features for classification, which is a standard approach in
1) Recursive autoconvolution and ICA or PCA: To demonunsupervised learning [6, 8, 18]. For example, in case of three
layers, features of bottom layers (l = 1, 2) are pooled so that strate generalizability of our method, we investigated if
their spatial size equals the size of the third layer features, recursive autoconvolution is able to improve other learning
methods. For this purpose, we trained K1 = 256 filters with
afterwards they are concatenated.
In all cases, except for AutoCNN-L32, a linear SVM is used principal (PCA) and independent (ICA) component analysis
[21] on patches with n = 0 (Raw) and n = [0, 3] (RA) using
as a classifier.
the same procedure as with k-means (Section IV-A). Filter
B. Effect of filter size and number of filters
sizes have to be chosen based on results from the previous
First, we experiment with a simple single layer architecture experiment (Fig. 4 (b)-(d)). However, we had to use larger filters
(AutoCNN-S1) to see the effect of recursive autoconvolution to satisfy s1 ×s1 ×3 ≥ K1 , since we did not learn overcomplete
(Eq. 2) on classification performance depending on the filter dictionaries. Even though RA does not improve PCA filters
size (s1 ) and the number of filters (K1 ) (Fig. 4 (b)-(d)). for some reasons, in case of ICA the gap between results

with and without RA is notable (Table Ia), which confirms the
effectiveness of our method.
C. Multilayer performance
We performed a series of experiments with convolutional
architectures that match or similar to the ones in the previous
works on unsupervised learning [18, 27]. It allows us to fairly
compare features and to see the benefit of RA for a multilayer
AutoCNN.
1) Comparison to Exemplar-CNN: Compared to ExemplarCNNs [18], our results appear to be slightly inferior at first
(Table II), but our networks have only 3 layers and we do
not exploit data augmentation so extensively. For instance, we
do not use any color augmentation (see details in Table VI).
To improve our results, we trained a larger 3 layer network
(AutoCNN-L), and were able to outperform Exemplar-CNNs
on CIFAR-10(400).
TABLE II: Comparison of our features with and without RA to Exemplar-CNN
[18] (A: data augmentation, A∗ : large data augmentation, average accuracy
(%) for 10 folds is reported).
Model
Exemplar-CNN +
Exemplar-CNN +

CIFAR-10(400)

STL-10

# filters

75.7
77.4

74.9
75.4

64-128-256-512
92-256-512-1024

A∗
A∗

AutoCNN-S
AutoCNN-S - Rootsift
AutoCNN-S + A
AutoCNN-M + A
AutoCNN-L + A

Raw

RA

Raw RA

66.5
60.9
69.9
72.2
75.9

68.4
61.0
72.1
74.5
77.6

60.6
51.9
67.6
69.7
72.4

62.3
52.5
69.2
70.6
73.1

64-128-256
64-128-256
64-128-256
92-256-512
256-1024-2048

instance, the same error was reported in [12] and 0.47% in
[11] with 2 and 3 layer convolutional networks respectively.
Without RA our model performs significantly worse.
TABLE III: Comparison of our features with and without RA to CONV-WTA
[27] on full CIFAR-10 (A: data augmentation).
Model

CIFAR-10

# filters

72.3
77.9
80.1

256
256-1024
256-1024-4096

CONV-WTA
CONV-WTA
CONV-WTA

Raw RA
AutoCNN-S1-256
AutoCNN-M2
AutoCNN-M2 + PCA4k
AutoCNN-L + PCA4k
AutoCNN-L + PCA4k + A

71.3
80.0
77.4
80.1
82.7

72.4
256
80.4
256-1024
78.9
256-1024
81.4 256-1024-2048
84.4 256-1024-2048

TABLE IV: Classification errors on MNIST using unsupervised methods (for
MNIST(100) the format is error ± std).
Model
Sparse coding [2]
C-SVDDNet [28]
CONV-WTA [27]
CONV-WTA [27]

MNIST (100)

MNIST

# filters

−
−
−
1.92

0.59
0.35
0.64
0.48

169
400
128
128-2048

0.82
0.69
0.57
0.39

128
128
128-1024
128-1024

AutoCNN-S1-128
2.89 ±
AutoCNN-S1-128 + RA 2.45 ±
AutoCNN-S2
1.91 ±
AutoCNN-S2 + RA
1.75 ±

0.16
0.10
0.10
0.10

Semi-supervised best 0.84 ± 0.08 [17]

0.5 [6]

D. Performance of large AutoCNN-L32
Note, however, that for all models in Table II the results with
Unsupervised layer-wise learning does not suffer from
RA are better than without (columns ’Raw’), while all model
overfitting,
but making layers wider than in our AutoCNN-L is
settings are kept identical. In fact, for CIFAR-10(400) the
computationally
demanding. Meanwhile, splitting features into
network without recursive autoconvolution has to be virtually
G
groups
(see
Section
IV-A2) provides more features, and in
l
doubled in size in order to catch up wih the score of the network
certain
settings
it
also
benefits
classification. So we designed
with RA (compare results of AutoCNN-S and AutoCNN-M).
a
large
AutoCNN-L32
network
with Gl = 32 groups. Due
Using a small AutoCNN-S network we estimated the
to
feature
splitting
and
the
idea
of shared filters introduced
contribution of Rootsift [26] to our results by removing it
earlier
in
this
work
(Section
IV-A3
and Table Ib), training
from our pipeline. Not only it boosts classification, but also
time
remains
comparable
to
AutoCNN-L.
We also tried other
allows recursive autoconvolution to work more efficiently for
architectures,
but
present
results
only
with
the best one.
multilayer models. However, note that our plots in Fig. 4 and
1) Dimension reduction and classification: Output features
all results on MNIST are obtained without it, so Rootsift is
of AutoCNN-L32 are prohibitively large, so we first apply
not always required.
2) Comparison to CONV-WTA: Compared to the Winner randomized principal component analysis [31] together with
Takes All Autoencoder (CONV-WTA) [27], our networks are whitening and then train an SVM on the projected data. In this
more efficient even with a smaller number of filters (Table III). case, we use an RBF-SVM in an efficient GPU implementation
With RA our results are further improved. In a few cases we [32]. A nonlinear SVM is chosen, because it performs better
had to project features with PCA to 4096 (PCA4k) dimensions with the projected features. The SVM regularization constant
before classification to fit all CIFAR-10 features into memory. It is fixed to C = 16 and the width of the RBF kernel is chosen
is an undesirable operation, because PCA degrades our features. to be γ = 1/NP CA , where NP CA is the dimensionality of the
We also compare our models to CONV-WTA on projected feature vector, which is the input of the SVM.
MNIST(100) and full MNIST (Table IV). Two layer AutoCNN2) Evaluation: AutoCNN-L32 achieves very competitive
S2 with recursive autoconvolution yields better results than results on both reduced and full datasets (Table V). We found
CONV-WTA, even though our model has two times fewer filters that for layers 2-3 of this particular model it is better to train
in the second layer (other parameters are the same, Rootsift is filters with PCA rather than with k-means. It is surprising,
not used on MNIST). Our error of 0.39% on full MNIST is because according to our results with a single layer model
competitive even when compared to supervised learning. For (Table Ia), PCA should be worse than k-means in this task. But

TABLE V: Classification accuracies on the test sets using unsupervised methods (A: data augmentation, † : obtained with a larger network and supervised
learning of connections between layers).
Model

CIFAR-10(400)

CIFAR-10

STL-10

CIFAR-100

# filters

NOMP-20 [8]
Conv. clustering, unsup. [6]
CONV-WTA [27]
Committee of nets + A [7]
k-means + A [20]
Exemplar-CNN + A∗ [18]

72.2 ± 0.4
−
−
−
72.6 ± 0.7
77.4 ± 0.2

82.9
−
80.1
−
81.9
84.3

67.9 ± 0.6
65.4 (74.1† )
−
68.0 ± 0.6
63.7
75.4 ± 0.3

60.8
−
−
−
−
−

3200-6400-6400
96-1536
256-1024-4096
300-5625
3 layers, 6400
92-256-512-1024

AutoCNN-L32 + RA + PCA1k
AutoCNN-L32 + A + PCA1.5k
AutoCNN-L32 + RA + A + PCA1.5k

76.4 ± 0.4
78.2 ± 0.3
79.4 ± 0.3

85.4
87.1
87.9

68.7 ± 0.5
74.0 ± 0.6
74.5 ± 0.6

63.9
67.1
67.8

1024-256-1024

73.8 ± 0.4 [29]
79.6 ± 0.5 [17]

89.1 [30]
92.2 [13]

−
74.3 [13]

67.7 [30]
69.1 [13]

Pretrained on ImageNet
Semi-supervised state of the art

we obtain just 77.4 ± 0.3% on CIFAR-10(400) with AutoCNNL32 if k-means is used for all layers. To obtain results on
CIFAR-100 we use the same settings as on CIFAR-10.

In this work, we show good results both for simple and more
complex datasets, as well as both for smaller and larger ones
using the same model with few tuning parameters. Among
unsupervised learning methods without data augmentation we
E. Analysis of results
report state of the art results for all datasets. One of the main
Notably, in most of the previous works (see Tables IV and reasons for such results is that recursive autoconvolution allows
V), the classification results are very good either for simple us to learn a set of filters with rich spatio-frequency properties.
grayscale images (MNIST) or more complex colored datasets In AutoCNNs, most of the learned filters have joint spatial
(CIFAR-10, CIFAR-100, STL-10) or for smaller or larger and frequency resolution close to the theoretical minimum
datasets only. The only exception seems to be Ladder Networks (Fig. 4(a), RA with n = [0, 3]), i.e. they resemble simple edge
[17], which are much larger and deeper than our models and detectors, such as Gabor filters. Other filters are uniformly
are semi-supervised. Also, in [28] better results are achieved distributed across a wide range of resolutions and typically
on MNIST, but it can be quite easy to fine tune to such a have complex meaningful shapes (Fig. 1, 2). By learning large
simple task 2 . Our models are especially effective for CIFAR- sets of such filters our models are able to detect a lot of
10, showing performance close to Ladder Networks. Our results highly diverse features, so that an SVM can easily choose the
on this dataset (both reduced and full) are several percent higher discriminative ones.
(absolute difference) compared to other unsupervised models.
a) Computational complexity.: Note that the numbers of
On full CIFAR-10 and CIFAR-100 with 5000 and 500 training network filters presented in Tables II-V do not always reflect
labels per class respectively, our models also outperform the total number of trainable parameters (N ) nor the total
some advanced fully supervised CNNs [11, 12] and better computational cost. While our 3 layer AutoCNN-L32 model
or comparable to CNNs pretrained in an unsupervised way has more filters than some CNNs, for convolutional layers it
[16] or on the large-scale dataset with over 1 million of training has N ≈ 6.8 · 106 parameters, while both the largest Exemplarsamples [29, 30]. On STL-10 our network with RA compares CNN [18] and CONV-WTA [27] models have ≈ 40 · 106
favorably to the 10 layer convolutional autoencoder (SWWAE) parameters (in case of CIFAR-10). Training our large 3 layer
[13]. Other previous works, showing higher accuracies, are network on full CIFAR-10 with data augmentation takes about
either fully or semi-supervised.
85 minutes on NVIDIA GTX 980 Ti, Intel Xeon CPU E5Our deeper networks outperform our shallow ones, which 2620v3 and 64GB RAM in a Matlab implementation with
suggests the importance of depth in our case in the same way VLFeat [33], MatConvNet [34] and GTSVM [32]. It’s also
as in supervised CNNs.
important that computational cost of recursive autoconvolution
One of the drawbacks in our work is that our best results for filter learning is less than 10% of overall training time.
on CIFAR and STL-10 are obtained with a nonlinear SVM,
Source codes to reproduce our results are available at
but we believe that given the overall simplicity of our models https://github.com/bknyaz/autocnn_unsup.
it can be reasonable to use a more complex kernel. Moreover,
VI. C ONCLUSION
in our experience the RBF kernel does not always improve
The importance of unsupervised learning in visual tasks
classification results if used on top of CNN features, but does
is
increasing and development is driven by the necessity to
in our case. In addition, we show competitive results with
better
exploit massive amounts of unlabeled data. We propose
a linear SVM in Tables Ia-IV. We also do not thoroughly
a
novel
idea for unsupervised feature learning and report
validate all model components (which would be out of scope),
competitive
results in several image classification tasks among
because the main intention of this work is to show that recursive
the
works
not
relying on supervised learning. Moreover, we
autoconvolution boosts classification in a number of different
report
state
of
the art results among unsupervised methods
settings and is important to achieve our best results.
without data augmentation. We adopt recursive autoconvolution
2 See https://github.com/bknyaz/gabors
and demonstrate its great utility for unsupervised learning

methods, such as k-means and ICA. We argue that it can also
be integrated into more recent learning methods, such as convolutional clustering, to boost their performance. Furthermore, we
significantly reduce the total number of trainable parameters by
using shared filters. As a result, the proposed autoconvolutional
network performs better than most of the unsupervised, and
several supervised, models in various classification tasks with
only few, but also with thousands of, labeled samples.
ACKNOWLEDGMENT
This work is jointly supported by the German Academic
Exchange Service (DAAD) and the Ministry of Education and
Science of the Russian Federation (project number 3708).
R EFERENCES
[1] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep
learning. Nature, 521(7553):436–444, 2015.
[2] Kai Labusch, Erhardt Barth, and Thomas Martinetz. Simple
method for high-performance digit recognition based on sparse
coding. Neural Networks, IEEE Transactions on, 19(11):1985–
1989, 2008.
[3] Adam Coates, Andrew Y Ng, and Honglak Lee. An analysis
of single-layer networks in unsupervised feature learning. In
International conference on artificial intelligence and statistics,
pages 215–223, 2011.
[4] Adam Coates and Andrew Y Ng. The importance of encoding
versus training with sparse coding and vector quantization. In
ICML, pages 921–928, 2011.
[5] Quoc V Le, Alexandre Karpenko, Jiquan Ngiam, and Andrew Y
Ng. ICA with reconstruction cost for efficient overcomplete
feature learning. In NIPS, pages 1017–1025, 2011.
[6] Aysegul Dundar, Jonghoon Jin, and Eugenio Culurciello. Convolutional clustering for unsupervised learning. arXiv preprint
arXiv:1511.06241v2, 2016.
[7] Bogdan Miclut, Thomas Kaester, Thomas Martinetz, and Erhardt
Barth. Committees of deep feedforward networks trained with
few data. arXiv preprint arXiv:1406.5947, 2014.
[8] Tsung-Han Lin and HT Kung. Stable and efficient representation
learning with nonnegativity constraints. Journal of Machine
Learning Research, 2014.
[9] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner.
Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.
[10] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers
of features from tiny images, 2009.
[11] Matthew D Zeiler and Rob Fergus. Stochastic pooling for
regularization of deep convolutional neural networks. arXiv
preprint arXiv:1301.3557, 2013.
[12] Julien Mairal, Piotr Koniusz, Zaid Harchaoui, and Cordelia
Schmid. Convolutional kernel networks. In NIPS, pages 2627–
2635, 2014.
[13] Junbo Zhao, Michael Mathieu, Ross Goroshin, and Yann
Lecun. Stacked what-where auto-encoders. arXiv preprint
arXiv:1506.02351, 2015.
[14] BA Knyazev and VM Chernenkiy. Convolutional sparse coding
for static and dynamic images analysis. Science & Education of
Bauman MSTU/Nauka i Obrazovanie of Bauman MSTU, 1(11),
2014.
[15] Matthew D Zeiler and Rob Fergus. Visualizing and understanding
convolutional networks. In European Conference on Computer
Vision, pages 818–833. Springer, 2014.
[16] Tom Le Paine, Pooya Khorrami, Wei Han, and Thomas S
Huang. An analysis of unsupervised pre-training in light of
recent advances. arXiv preprint arXiv:1412.6597, 2014.

[17] Antti Rasmus, Mathias Berglund, Mikko Honkala, Harri Valpola,
and Tapani Raiko. Semi-supervised learning with ladder
networks. In NIPS, pages 3532–3540, 2015.
[18] Alexey Dosovitskiy, Philipp Fischer, Jost Tobias Springenberg,
Martin Riedmiller, and Thomas Brox. Discriminative unsupervised feature learning with exemplar convolutional neural
networks. arXiv preprint arXiv:1406.6909v2, 2015.
[19] Xiaolong Wang and Abhinav Gupta. Unsupervised learning of
visual representations using videos. In Proceedings of the IEEE
International Conference on Computer Vision, pages 2794–2802,
2015.
[20] Ka Y Hui. Direct modeling of complex invariances for visual
object features. In ICML, 2013.
[21] Aapo Hyvärinen. Fast and robust fixed-point algorithms for
independent component analysis. Neural Networks, IEEE
Transactions on, 10(3):626–634, 1999.
[22] Hilton Bristow, Anders Eriksson, and Simon Lucey. Fast
convolutional sparse coding. In Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition, pages
391–398, 2013.
[23] V Dose, Th Fauster, and H-J Gossmann. The inversion of
autoconvolution integrals. Journal of Computational Physics,
41(1):34–50, 1981.
[24] Rudolf Gorenflo and Bernd Hofmann. On autoconvolution and
regularization. Inverse Problems, 1994.
[25] Janne Heikkila. Multi-scale auto-convolution for affine invariant
pattern recognition. In Pattern Recognition, 2002. Proceedings.
16th International Conference on, volume 1, pages 119–122.
IEEE, 2002.
[26] Relja Arandjelović and Andrew Zisserman. Three things
everyone should know to improve object retrieval. In Computer
Vision and Pattern Recognition (CVPR), 2012 IEEE Conference
on, pages 2911–2918. IEEE, 2012.
[27] Alireza Makhzani and Brendan J Frey. Winner-take-all autoencoders. In NIPS, pages 2773–2781, 2015.
[28] Dong Wang and Xiaoyang Tan. Unsupervised feature learning
with C-SVDDNet. arXiv preprint arXiv:1412.7259, 2014.
[29] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised
representation learning with deep convolutional generative
adversarial networks. arXiv preprint arXiv:1511.06434, 2015.
[30] Lars Hertel, Erhardt Barth, Thomas Käster, and Thomas Martinetz. Deep convolutional neural networks as generic feature
extractors. In 2015 International Joint Conference on Neural
Networks (IJCNN), pages 1–4. IEEE, 2015.
[31] Nathan Halko, Per-Gunnar Martinsson, and Joel A Tropp.
Finding structure with randomness: Probabilistic algorithms for
constructing approximate matrix decompositions. SIAM review,
53(2):217–288, 2011.
[32] Andrew Cotter, Nathan Srebro, and Joseph Keshet. A GPUtailored approach for training kernelized SVMs. In SIGKDD,
pages 805–813. ACM, 2011.
[33] Andrea Vedaldi and Brian Fulkerson. VLFeat: An open and
portable library of computer vision algorithms. In Proceedings
of the 18th ACM international conference on Multimedia, pages
1469–1472. ACM, 2010.
[34] Andrea Vedaldi and Karel Lenc. MatConvNet: Convolutional
neural networks for matlab. In Proceedings of the 23rd Annual
ACM Conference on Multimedia Conference, pages 689–692.
ACM, 2015.

TABLE VI: Dataset and some model parameters used in the experiments (LCN - Local Contrast Normalization)
Dataset

Preprocessing

LCN (>1 layers)

Multidictionary (>1 layers)

Data augmentation (A)

MNIST
CIFAR-10, CIFAR-100
STL-10

−
ZCA-whitening
ZCA-whitening

−
+
+

+
+
+

−
mirroring
mirroring, crops(72px), scaling(1-1.3), rotation(±10 degrees)

TABLE VII: Network architectures used in the experiments
Model

L K1 n1

s1 , d 1

R1

m1

G1 K2 n2

K1 1-3 s1 xs1 x1 |x|
128 0 7x7x3 |x|
128 1-3 11x11x3 |x|
128 1-3 7x7x3 |x|

4(4)
4(4)
4(4)
5(3)

− −
− −
− −
1 1024

s2 , d 2

R2 m2 G2 K3 n3

s3 , d 3

R3 m3 root SVM

MNIST
AutoCNN-S1
1
AutoCNN-S1-128
1
AutoCNN-S1-128 + RA 1
AutoCNN-S2
2

−
−
−
−
−
−
0-2 5x5x128

−
−
−
|x|

−
−
−
3(2)

−
−
−
−

−
−
−
−

−
−
−
−

−
−
−
−

−
−
−
−

−
−
−
−

−
−
−
−

Linear
Linear
Linear
Linear

−
−
−
7x7x4
5x5x256
5x5x64
5x5x92
5x5x256
5x5x32

−
−
−
|x|
|x|
|x|
|x|
|x|
|x|

−
−
−
5(4)
5(4)
3(2)
3(2)
3(2)
3(2)

−
−
−
−
−
1
1
1
32

−
−
−
−
−
256
512
2048
1024

−
−
−
−
−
0-2
0-2
0-2
0-2

−
−
−
−
−
5x5x128
5x5x256
5x5x1024
5x5x256

−
−
−
−
−
|x|
|x|
|x|
|x|

−
−
−
−
−
3(2)
3(2)
3(2)
3(2)

−
+
+
+
+
+
+
+
+

Linear
Linear
Linear
Linear
Linear
Linear
Linear
Linear
RBF

−
5x5x64
5x5x92
5x5x256
5x5x32

−
|x|
|x|
|x|
|x|

−
4(3)
4(3)
4(3)
4(3)

−
1
1
1
32

−
256
512
2048
1024

−
0-2
0-2
0-2
0-2

−
5x5x128
5x5x256
5x5x1024
5x5x256

−
|x|
|x|
|x|
|x|

−
3(2)
3(2)
3(2)
3(2)

−
+
+
+
+

Linear
Linear
Linear
Linear
RBF

CIFAR-10 and CIFAR-100
AutoCNN-S1
1 K1 0-3 s1 xs1 x3
AutoCNN-S1-256
1 256 0 9x9x3
AutoCNN-S1-256 + RA 1 256 0-3 13x13x3
AutoCNN-S32
2 128 0-3 9x9x3
AutoCNN-M2
2 256 0-3 5x5x3
AutoCNN-S
3 64 0-3 5x5x3
AutoCNN-M
3 92 0-3 5x5x3
AutoCNN-L
3 256 0-3 5x5x3
AutoCNN-L32
3 1024 0-3 5x5x3

relu 8(8)
relu 8(8)
relu 8(8)
relu 3(2)
relu 3(2)
relu 3(2)
relu 3(2)
relu 3(2)
relu 3(2)

−
−
−
32
1
1
1
1
32

−
−
−
64
1024
128
256
1024
256

−
−
−
0-2
0-2
0-3
0-3
0-3
0-2

STL-10
AutoCNN-S1
AutoCNN-S
AutoCNN-M
AutoCNN-L
AutoCNN-L32

1 K1
3 64
3 92
3 256
3 1024

0-3 s1 xs1 x3
0-3 7x7x3
0-3 7x7x3
0-3 7x7x3
0-3 7x7x3

relu 20(20) − − −
relu 5(4) 1 128 0-3
relu 5(4) 1 256 0-3
relu 5(4) 1 1024 0-3
relu 5(4) 32 256 0-2

L - number of layers;
Kl - number of filters in layer l;
sl , dl - filter size and depth for layer l;
nl - recursive autoconvolution (RA) order to learn filters for layer l (in case RA is not applied nl = 0);
ml - max pooling size and stride (in parentheses); for STL-10 in case of data augmentation ml = 4(3) instead of 5(4) in the first layer;
Gl - number of feature map groups;
Rl - rectifier after layer l (relu: max(0, x), |x| - absolute
p values);
root - Rootsift normalization after each layer (sign(x) |x|/kxk1 ).

