Accepted as a workshop contribution at ICLR 2015

A G ENERATIVE M ODEL
L EARNING

FOR

D EEP C ONVOLUTIONAL

arXiv:1504.04054v1 [stat.ML] 15 Apr 2015

Yunchen Pu, Xin Yuan and Lawrence Carin
Department of Electrical and Computer Engineering, Duke University, Durham, NC, 27708, USA
{yunchen.pu,xin.yuan,lcarin}@duke.edu

A BSTRACT
A generative model is developed for deep (multi-layered) convolutional dictionary
learning. A novel probabilistic pooling operation is integrated into the deep model,
yielding efficient bottom-up (pretraining) and top-down (refinement) probabilistic
learning. Experimental results demonstrate powerful capabilities of the model
to learn multi-layer features from images, and excellent classification results are
obtained on the MNIST and Caltech 101 datasets.

1

I NTRODUCTION

We develop a deep generative statistical model, which starts at the highest-level features, and maps
these through a sequence of layers, until ultimately mapping to the data plane (e.g., an image). The
feature at a given layer is mapped via a multinomial distribution to one feature in a block of features
at the layer below (and all other features in the block at the next layer are set to zero). This is
analogous to the method in Lee et al. (2009), in the sense of imposing that there is at most one
non-zero activation within a pooling block. We use bottom-up pretraining, in which initially we
sequentially learn parameters of each layer one at a time, from bottom to top, based on the features
at the layer below. However, in the refinement phase, all model parameters are learned jointly, topdown. Each consecutive layer in the model is locally conjugate in a statistical sense, so learning
model parameters may be readily performed using sampling or variational methods.

2

M ODELING F RAMEWORK

Assume N gray-scale images {X(n) }n=1,N , with X(n) ∈ RNx ×Ny ; the images are analyzed jointly
to learn the convolutional dictionary {D(k) }k=1,K . Specifically consider the model
K
X
X(n) =
D(k) ∗ (Z(n,k) W(n,k) ) + E(n) ,
(1)
k=1

where ∗ is the convolution operator, denotes the Hadamard (element-wise) product, the elements
of Z(n,k) are in {0, 1}, the elements of W(n,k) are real, and E(n) represents the residual. Z(n,k)
indicates which shifted version of D(k) is used to represent X(n) .
Assume an L-layer model, with layer L the top layer, and layer 1 at the bottom, closest to the
data. In the pretraining stage, the output of layer l is the input to layer l + 1, after pooling. Layer
l ∈ {1, . . . , L} has Kl dictionary elements, and we have:

PKl+1
(kl+1 ,l+1)
X(n,l+1) =
∗ Z(n,kl+1 ,l+1) W(n,kl+1 ,l+1) + E(n,l+1)
(2)
kl+1 =1 D


P
Kl
(kl ,l)
∗ Z(n,kl ,l) W(n,kl ,l) +E(n,l)
X(n,l) =
(3)
kl =1 D
|
{z
}
=S(n,kl ,l)

The expression X(n,l+1) may be viewed as a 3D entity, with its kl -th plane defined by a “pooled”
version of S(n,kl ,l) .
The 2D activation map S(n,kl ,l) is partitioned into nx × ny dimensional contiguous blocks (pooling
blocks with respect to layer l + 1 of the model); see the left part of Figure 1. Associated with each
1

Accepted as a workshop contribution at ICLR 2015

S (n,k2 ,2)

S (n,k2 ,2)

D
X (n,2)

0.3

(k2 ,2)

D (k2 ,2)
X (n,2)

0

0.9

0.8

0.3
0.9

0

0

0.3

0
0

0

0
0
0.9

0

1
0

0
0
0

0

S (n,k1,1)

0
0

D

(k1 ,1)

0.8

0

1

0
1

0

0
0.8
0

0

0

0
0

0

1

0

Z (n,k1,1)
S (n,k1,1)

0.1

0.3

0
0
0

0

0
0
0.9

0

0.1

0

0.8
0

0
0

D (k1,1)

X (n,1)

X (n,1)

Figure 1: Schematic of the proposed generative process. Left: bottom-up pretraining, right: top-down refinement. (Zoom-in for best visulization and a larger version can be found in the Supplementary Material.)
block of pixels in S(n,kl ,l) is one pixel at layer kl of X(n,l+1) ; the relative locations of the pixels
in X(n,l+1) are the same as the relative locations of the blocks in S(n,kl ,l) . Within each block of
S(n,kl ,l) , either all nx ny pixels are zero, or only one pixel is non-zero, with the position of that pixel
selected stochastically via a multinomial distribution. Each pixel at layer kl of X(n,l+1) equals the
largest-amplitude element in the associated block of S(n,kl ,l) (i.e., max pooling).
The learning performed with the top-down generative model (right part of Fig. 1) constitutes a
refinement of the parameters learned during pretraining, and the excellent initialization constituted
by the parameters learned during pretraining is key to the subsequent model performance.
In the refinement phase, we now proceed top down, from (2) to (3). The generative process constitutes D(kl+1 ,l+1) and Z(n,kl+1 ,l+1) W(n,kl+1 ,l+1) , and after convolution X(n,l+1) is manifested;
the E(n,l) is now absent at all layers, except layer l = 1, at which the fit to the data is performed.
Each element of X(n,l+1) has an associated pooling block in S(n,kl ,l) .

3 E XPERIMENTAL R ESULTS
We here apply our model to the MNIST and Caltech 101 datasets.
Table 1: Classification Error of MNIST data
MNIST Dataset Table 1 summaries the clasTest error
Methods
sification results of our model compared with
6-layer Conv. Net + 2-layer Classifier
0.35%
some related results, on the MNIST data. The
+ elastic distortions Ciresan et al. (2011)
second (top) layer features corresponding to
MCDNN Ciresan et al. (2012)
0.23%
the refined dictionary are sent to a nonlinear
SPCNN Zeiler & Fergus (2013)
0.47%
support vector machine (SVM) (Chang & Lin,
HBP Chen et al. (2013),
0.89%
2011) with Gaussian kernel, in a one-vs-all
2-layer cFA + 2-layer features
Ours, 2-layer model + 1-layer features
0.42%
multi-class classifier, with classifier parameters
tuned via 5-fold cross-validation (no tuning on the deep feature learning).
Caltech 101 Dataset We next consider the
Caltech 101 dataset.For Caltech 101 classification, we follow the setup in Yang et al.
(2009), selecting 15 and 30 images per category for training, and testing on the rest.
The features of testing images are inferred
based on the top-layer dictionaries and sent
to a multi-class SVM; we again use a Gaussian kernel non-linear SVM with parameters
tuned via cross-validation. Ours and related
results are summarized in Table 2.

Table 2: Classification Accuracy Rate of Caltech-101.
# Training Images per Category
DN Zeiler et al. (2010)
CBDN Lee et al. (2009)
HBP Chen et al. (2013)
ScSPM Yang et al. (2009)
P-FV Seidenari et al. (2014)
R-KSVD Li et al. (2013)
Convnet Zeiler & Fergus (2014)
Ours, 2-layer model + 1-layer features
Ours, 3-layer model + 1-layer features

15
58.6 %
57.7 %
58%
67 %
71.47%
79 %
83.8 %
70.02%
75.24%

30
66.9%
65.4%
65.7%
73.2%
80.13%
83%
86.5%
80.31%
82.78%

4 C ONCLUSIONS
A deep generative convolutional dictionary-learning model has been developed within a Bayesian
setting. The proposed framework enjoys efficient bottom-up and top-down probabilistic inference.
A probabilistic pooling module has been integrated into the model, a key component to developing a
principled top-down generative model, with efficient learning and inference. Extensive experimental
results demonstrate the efficacy of the model to learn multi-layered features from images.
2

Accepted as a workshop contribution at ICLR 2015

R EFERENCES
Chang, C.-C. and Lin, C.-J. LIBSVM: A library for support vector machines. ACM Transactions on Intelligent
Systems and Technology, 2011.
Chen, B., Polatkan, G., Sapiro, G., Blei, D., Dunson, D., and Carin, L. Deep learning with hierarchical
convolutional factor analysis. IEEE T-PAMI, 2013.
Ciresan, D., Meier, U., and Schmidhuber, J. Multi-column deep neural networks for image classification. In
CVPR, 2012.
Ciresan, D. C., Meier, U., Masci, J., Gambardella, L. M., and Flexible, J. Schmidhuber. high performance
convolutional neural networks for image classification. IJCAI, 2011.
Lee, H., Grosse, R., Ranganath, R., and Ng, A. Y. Convolutional deep belief networks for scalable unsupervised
learning of hierarchical representations. ICML, 2009.
Li, Q., Zhang, H., Guo, J., Bhanu, B., and An, L. Reference-based scheme combined with K-svd for scene
image categorization. IEEE Signal Processing Letters, 2013.
Seidenari, L., Serra, G., Bagdanov, A., and Del Bimbo, A. Local pyramidal descriptors for image recognition.
IEEE T-PAMI, 2014.
Yang, J., Yu, K., Gong, Y., and Huang, T. Linear spatial pyramid matching using sparse coding for image
classification. In CVPR, 2009.
Zeiler, M. and Fergus, R. Stochastic pooling for regularization of deep convolutional neural networks. ICLR,
2013.
Zeiler, M. and Fergus, R. Visualizing and understanding convolutional networks. ECCV, 2014.
Zeiler, M., Kirshnan, D., Taylor, G., and Fergus, R. Deconvolutional networks. CVPR, 2010.

3

