Semi-Supervised Learning with the Deep Rendering Mixture Model
Tan Nguyen1,2

Wanjia Liu1
Ethan Perez1
Richard G. Baraniuk1
Ankit B. Patel1,2
1
2
Rice University
Baylor College of Medicine
6100 Main Street, Houston, TX 77005
1 Baylor Plaza, Houston, TX 77030

arXiv:1612.01942v1 [stat.ML] 6 Dec 2016

{mn15, wl22, ethanperez, richb}@rice.edu

Abstract

ankitp@bcm.edu

recognition and image segmentation. However, DCNs are
still far behind humans in semi-supervised learning tasks,
in which only a few labels are available. The main difficulty in semi-supervised learning in DCNs is that, until
recently, there has not been a mathematical framework for
deep learning architectures. As a result, it is not clear how
DCNs encode the data distribution, making combining supervised and unsupervised learning challenging.

Semi-supervised learning algorithms reduce the high
cost of acquiring labeled training data by using both labeled and unlabeled data during learning. Deep Convolutional Networks (DCNs) have achieved great success in
supervised tasks and as such have been widely employed
in the semi-supervised learning. In this paper we leverage the recently developed Deep Rendering Mixture Model
(DRMM), a probabilistic generative model that models latent nuisance variation, and whose inference algorithm
yields DCNs. We develop an EM algorithm for the DRMM
to learn from both labeled and unlabeled data. Guided
by the theory of the DRMM, we introduce a novel nonnegativity constraint and a variational inference term. We
report state-of-the-art performance on MNIST and SVHN
and competitive results on CIFAR10. We also probe deeper
into how a DRMM trained in a semi-supervised setting represents latent nuisance variation using synthetically rendered images. Taken together, our work provides a unified framework for supervised, unsupervised, and semisupervised learning.

Recently, the Deep Rendering Mixture Model (DRMM)
[13, 14] has been developed as a probabilistic graphical
model underlying DCNs. The DRMM is a hierarchical generative model in which the image is rendered via multiple
levels of abstraction. It has been shown that the bottomup inference in the DRMM corresponds to the feedforward
propagation in the DCNs. The DRMM enables us to perform semi-supervised learning with DCNs. Some preliminary results for semi-supervised learning with the DRMM
are provided in [14]. Those results are promising, but
more work is needed to evaluate the algorithms across many
tasks.
In this paper, we systematically develop a semisupervised learning algorithm for the Non-negative DRMM
(NN-DRMM), a DRMM in which the intermediate rendered templates are non-negative. Our algorithm contains a
bottom-up inference pass to infer the nuisance variables in
the data and a top-down pass that performs reconstruction.
We also employ variational inference and the non-negative
nature of the NN-DRMM to derive two new penalty terms
for the training objective function. An overview of our algorithm is given in Figure 1. We validate our methods by
showing state-of-the-art semi-supervised learning results on
MNIST and SVHN, as well as comparable results to other
state-of-the-art methods on CIFAR10. Finally, we analyze
the trained model using a synthetically rendered dataset,
which mimics CIFAR10 but has ground-truth labels for nuisance variables, including the orientation and location of the
object in the image. We show how the trained NN-DRMM
encodes nusiance variations across its layers and show a
comparison against traditional DCNs.

1. Introduction
Humans are able to learn from both labeled and unlabeled data. Young infants can acquire knowledge about the
world and distinguish objects of different classes with only
a few provided “labels”. Mathematically, this poverty of input implies that the data distribution p(I) contains information useful for inferring the category posterior p(c|I). The
ability to extract this useful hidden knowledge from the data
in order to leverage both labeled and unlabeled examples
for inference and learning, i.e. semi-supervised learning,
has been a long-sought after objective in computer vision,
machine learning and computational neuroscience.
In the last few years, Deep Convolutional Networks
(DCNs) have emerged as powerful supervised learning
models that achieve near-human or super-human performance in various visual inference tasks, such as object
1

A

B

𝐸↓ : Top-Down
Reconstruction

C

Classification Cross Entropy, ℒ𝐻

K-L Distance, ℒ𝐾𝐿

𝜇𝑐
1
3

Update 𝜇𝑐

4

2

ON
OFF

Λ𝑔(𝐿)
Total Loss, ℒ
Update Λ𝑔(ℓ)

1
3

2
4

Λ𝑔(ℓ)
Rendering

Non-Negativity Constraint, ℒ𝑁𝑁

Reconstruction Error, ℒ𝑅𝐶

Λ𝑔(1)

Noise

𝐸↑ : Bottom-Up
Inference

Figure 1: Semi-supervised learning for Deep Rendering Mixture Model (DRMM) (A) Computation flow for semi-supervised DRMM
loss function and its components. Dashed arrows indicate parameter update. (B) The Deep Rendering Mixture Model (DRMM). All
dependence on pixel location x has been suppressed for clarity. (C) DRMM generative model: a single super pixel x(`) at level ` (green,
upper) renders down to a 3 × 3 image patch at level ` − 1 (green, lower), whose location is specified by g (`) (light blue). (C) shows only
the transformation from level ` of the hierarchy of abstraction to level ` − 1.

2. Related Work
We focus our review on semi-supervised methods that
employ neural network structures and divide them into different types.
Autoencoder-based Architectures: Many early works in
semi-supevised learning for neural networks are built upon
autoencoders [1]. In autoencoders, the images are first
projected onto a low-dimensional manifold via an encoding
neural network and then reconstructed using a decoding
neural network. The model is learned by minimizing the
reconstruction error. This method is able to learn from
unlabeled data and can be combined with traditional neural
networks to perform semi-supervised learning. In this
line of work are the Contractive Autoencoder [16], the
Manifold Tangent Classifier, the Pseudo-label Denoising
Auto Encoder [9], the Winner-Take-All Autoencoders
[11], and the Stacked What-Where Autoencoder [20].
These architectures perform well when there are enough

labels in the dataset but fail when the number of labels
is reduced since the data distribution is not taken into
account. Recently, the Ladder Network [15] was developed
to overcome this shortcoming. The Ladder Network
approximates a deep factor analyzer where each layer in the
model is a factor analyzer. Deep neural networks are then
used to do approximate bottom-up and top-down inference.
Deep Generative Models: Another line of work in semisupervised learning is to use neural networks to estimate
the parameters of a probabilistic graphical model. This
approach is applied when the inference in the graphical
model is hard to derive or when the exact inference is
computationally intractable. The Deep Generative Model
family is in this line of work [7, 10].
Both Ladder Networks and Deep Generative Models
yield good semi-supervised learning performance on

Q

(L)

benchmarks.
They are complementary to our semisupervised learning on DRMM. However, our method is
different from these approaches in that the DRMM is the
graphical model underlying DCNs, and we theoretically
derive our semi-supervised learning algorithm as a proper
probabilistic inference against this graphical model.

Λg(`) µc(L) represents the sequence of local nuisance
transformations that partially render finer-scale details as
we move from abstract to concrete. Note that the factorized
structure of the DRMM results in an exponential reduction
in the number of free parameters. This enables efficient
inference, learning, and better generalization.

Generative Adversarial Networks (GANs): In the last
few years a variety of GANs have achieved promising results in semi-supervised learning on different benchmarks,
including MNIST, SVHN and CIFAR10. These models also
generate good-looking images of natural objects. In GANs,
two neural networks play a minimax game. One generates images, and the other classifies images. The objective
function is the game’s Nash equilibrium, which is different
from standard object functions in probabilistic modeling. It
would be both exciting and promising to extend the DRMM
objective to a minimax game as in GANs, but we leave this
for future work.

A useful variant of the DRMM is the Non-Negative
Deep Rendering Mixture Model (NN-DRMM), where the
intermediate rendered templates are constrained to be nonnegative. The NN-DRMM model can be written as

3. Deep Rendering Mixture Model

`

(`+1)

zn(`) = Λ

(`+1)

gn

···Λ

(L)
(L)

gn

µc(L) ≥ 0
n

∀` ∈ {1, . . . , L}. (4)

It has been proven that the inference in the NN-DRMM
via a dynamic programming algorithm leads to the feedforward propagation in a DCN. This paper develops a
semi-supevised learning algorithm for the NN-DRMM. For
brevity, throughout the rest of the paper we will drop the
NN.

The Deep Rendering Mixture Model (DRMM) is a
recently developed probabilistic generative model whose
bottom-up inference, under a non-negativity assumption,
is equivalent to the feedforward propagation in a DCN
[13, 14]. It has been shown that the inference process in
the DRMM is efficient due to the hierarchical structure of
the model. In particular, the latent variations in the data are
captured across multiple levels in the DRMM. This factorized structure results in an exponential reduction of the free
parameters in the model and enables efficient learning and
inference. The DRMM can potentially be used for semisupervised learning tasks [13].

Sum-Over-Paths Formulation of the DRMM: The
DRMM can be can be reformulated by expanding out
the matrix multiplications in the generation process into
scalar products.
Then each pixel intensity Ix =
P (L) (L)
(1) (1)
is the sum over all active paths
p λp ap · · · λp ap
leading to that pixel of the product of weights along that
path. The sparsity of a controls the number fraction of active paths. Figure 2 depicts the sum-over-paths formulation
graphically.

Definition 1 (Deep Rendering Mixture Model). The Deep
Rendering Mixture Model (DRMM) is a deep Gaussian Mixture Model (GMM) with special constraints on the latent
variables. Generation in the DRMM takes the form:

Our semi-supervised learning algorithm for the DRMM
is analogous to the hard Expectation-Maximization (EM)
algorithm for GMMs [2, 13, 14]. In the E-step, we perform
a bottom-up inference to estimate the most likely joint configuration of the latent variables ĝ and the object category ĉ
given the input image. This bottom-up pass is then followed
by a top-down inference which uses (ĉ, ĝ) to reconstruct the
image Iˆ ≡ µĉĝ and compute the reconstruction error LRC .
It is known that when applying a hard EM algorithm on
GMMs, the reconstruction error averaged over the dataset is
proportional to the expected complete-data log-likelihood.
For labeled images, we also compute the cross-entropy LH
between the predicted object classes and the given labels
as in regular supervised learning tasks. In order to further
improve the performance, we introduce a Kullback-Leibler
divergence penalty LKL on the predicted object class ĉ and
a non-negativity penalty LN N on the intermediate rendered
(`)
templates zn at each layer into the training cost objective
function. The motivation and derivation for these two terms

c(L) ∼ Cat({πc(L) }),
µcg ≡ Λg µc(L) ≡
2

g (`) ∼ Cat({πg(`) })

(1)
(2)
Λg(1) Λg(2)

I ∼ N (µcg , σ 1D(0) ),

(L−1) (L)
. . . Λg(L−1) Λg(L) µc(L)

(1)
(2)
(3)

where ` ∈ [L] ≡ {1, 2, . . . , L} is the layer, c(L) is the object
category, g (`) are the latent (nuisance) variables at layer `,
(`)
and Λg(`) are parameter dictionaries that contain templates
at layer `. Here the image I is generated by adding isotropic
Gaussian noise to a multiscale “rendered” template µcg .
In the DRMM, the rendering path is defined as the
sequence (c(L) , g (L) , . . . , g (`) , . . . , g (1) ) from the root
(overall class) down to the individual pixels at ` = 0. The
variable µcg is the template used to render the image, and

4. DRMM-based Semi-Supervised Learning
4.1. Learning Algorithm

Figure 2: (A)The Sum-over-Paths Formulation of the DRMM. Each rendering path contributes only if it is active (green)[14]. While
exponentially many possible rendering paths exist, only a very small fraction are active. (B) Rendering from layer ` → ` − 1 in the
DRMM. (C) Inference in the Nonnegative DRMM leads to processing identical to the DCN.

are discussed in section 4.2 and 4.3 below. The final objective function for semi-supervised learning in the DRMM is
given by L ≡ αH LH +αRC LRC +αKL LKL +αN N LN N ,
where αCE , αRC , αKL and αN N are the weights for the
cross-entropy loss LCE , reconstruction loss LRC , variational inference loss LKL , and the non-negativity penalty
loss LN N , respectively. The losses are defined as follows:
1 X X
LH ≡ −
[ĉn = cn ] log q (c|In )
(5)
|DL |
n∈DL c∈C

LRC ≡
LKL

1
N

N
X

In − Iˆn

n=1

2

(6)

2



N
1 XX
q (c|In )
≡
q (c|In ) log
N n=1
p(c)

During training, instead of a closed-form M step as in
EM algorithm for GMMs, we use gradient-based optimization methods such as stochastic gradient descent to optimize
the objective function.

4.2. Variational Inference
The DRMM can compute the most likely latent configuration (ĉn , ĝn ) given the image In , and therefore, allows the exact inference of p(c, ĝn |In ). Using variational inference, we would like the approximate posterior
q(c, ĝn |In ) ≡ maxp(c, g|In ) to be close to the true posteg∈G
P
rior p(c|I) =
p(c, g|In ) by minimizing the KL diverg∈G

(7)

gence D (q(c, ĝn |In )kp(c|In )). It has been shown in [3]
that this optimization is equivalent to the following:

c∈C

LN N

N
L

1 XX
≡
max 0, −zn`
N n=1

2
2

min −Eq [ln p(In |z)] + DKL (q(z|In )||p(z)),
.

q

(8)

`=1

Here, q (c|In ) is an approximation of the true posterior
p(c|In ). In the context of the DRMM and the DCN, q (c|In )
is the SoftMax activations, p(c) is the class prior, DL is the
set of labeled images, and [ĉn = cn ] = 1 if ĉn = cn and
0 otherwise. The max{0, ·} operator is applied elementwise and equivalent to the ReLu activation function used in
DCNs.

(9)

where z = c. A similar idea has been employed in variational autoencoders [6], but here instead of using a Gaussian
distribution, z is a categorical random variable. An extension of the optimization 9 is given by:
min −Eq [ln p(In |z)] + βDKL (q(z|In )||p(z)).
q

(10)

As has been shown in [4], for this optimization, there exists a value for β such that latent variations in the data are

optimally disentangled.
The KL divergence in Eqns. 9 and 10 results in the LKL
loss in the semi-supervised learning objective function for
DRMM (see Eqn. 5). Similarly, the expected reconstruction error −Eq [ln p(In |z)] corresponds to the LRC loss in
the objective function. Note that this expected reconstruction error can be exactly computed for the DRMM since
there are only a finite number of configurations for the class
c. When the number of object classes is large, such as in
ImageNet [17] where there are 1000 classes, sampling techniques can be used to approximate −Eq [ln p(In |z)]. From
our experiments (see Section 5), we notice that for semisupervised learning tasks on MNIST, SVHN, and CIFAR10,
using the most likely ĉ predicted in the bottom-up inference
to compute the reconstruction error yields the best classification accuracies.

4.3. Non-Negativity Constraint Optimization
In order to derive the DCNs from the DRMM, the intermediate rendered templates zn` must be non-negative[14].
This is necessary in order to apply the max-product algorithm, wherein we can push the max to the right to get
an efficient message passing algorithm. We enforce this
condition in the Top-Down inference of the DRMM by introducing new non-negativity constrains zn` ≥ 0 , ∀` ∈
{1, . . . , L} into the optimization 9 and 10. There are various well-developed methods to solve optimization problems with non-negativity constraints. We employ a simple but useful approach, which adds an extra non-negativity

PN PL
2
penalty, in this case, N1 n=1 `=1 max 0, −zn` 2 ,
into the objective function. This yields an unconstrained
optimization which can be solved by gradient-based methods such as stochastic gradient descent. We cross-validate
the penalty weight αN N .

5. Experiments
We evaluate our methods on the MNIST, SVHN, and
CIFAR10 datasets. In all experiments, we perform semisupervised learning using the DRMM with the training objective including the cross-entropy cost, the reconstruction
cost, the KL-distance, and the non-negativity penalty discussed in Section 4.1. We train the model on all provided training examples with different numbers of labels
and report state-of-the-art test errors on MNIST and SVHN.
The results on CIFAR10 are comparable to state-of-the-art
methods. In order to focus on and better understand the
impact of the KL and NN penalties on the semi-supervised
learning in the DRMM, we don’t use any other regularization techniques such as DropOut or noise injection in our
experiments. We also only use a simple stochastic gradient
descent optimization with exponentially-decayed learning
rates to train the model. Applying regularization and using better optimization methods like ADAM [6] may help

improve the semi-supervised learning performance of the
DRMM. More model and training details are provided in
the Appendix.

5.1. MNIST
MNIST dataset contains 60,000 training images and
10,000 test images of handwritten digits from 0 to 9. Each
image is of size 28-by-28. For evaluating semi-supervised
learning, we randomly choose NL ∈ {50, 100, 1000} images with labels from the training set such that the amounts
of labeled training images from each class are balanced.
The remaining training images are provided without labels.
We use a 5-layer DRMM with the feedforward configuration similar to the Conv Small network in [15]. We apply
batch normalization on the net inputs and use stochastic gradient descent with exponentially-decayed learning rate to
train the model.
Table 1 shows the test error for each experiment. The KL
and NN penalties help improve the semi-supervised learning performance across all setups. In particular, the KL
penalty alone reduces the test error from 13.41% to 1.36%
when NL = 100. Using both the KL and NN penalties,
the test error is reduced to 0.57%, and the DRMM achieves
state-of-the-art results in all experiments. 1 We also analyze the value that the KL and NN penalties add to the
learning. Table 2 reports the reductions in test errors for
NL ∈ {50, 100, 1K} when using the KL penalty only,
the NN penalty only, and both penalties during training.
Individually, the KL penalty leads to significant improvements in test errors (12.05% reduction in test error when
NL = 100), likely since it helps disentangle latent variations in the data. In fact, for a model with continuous latent
variables, it has been experimentally shown that there exists
an optimal value for the KL penalty αKL such that all of
the latent variations in the data are almost optimally disentangled [4]. More results are provided in the Appendix.

5.2. SVHN
Like MNIST, the SVHN dataset is used for validating
semi-supervised learning methods. SVHN contains 73,257
color images of street-view house number digits. For training, we use a 9-layer DRMM with the feedforward propagation similar to the Conv Large network in [15]. Other
training details are the same as for MNIST. We train our
model on NL = 500, 1K, 2K and show state-of-the-art results in Table 3.
1 The

results for improved GAN is on permutation invariant MNIST
task while the DRMM performance is on the regular MNIST task. Since
the DRMM contains local latent variables t and a at each level, it is not
suitable for tasks such as permutation invariant MNIST

Model
DGN [7]
catGAN [19]
Virtual Adversarial [12]
Skip Deep Generative Model [10]
LadderNetwork [15]
Auxiliary Deep Generative Model [10]
ImprovedGAN [18]
DRMM 5-layer
DRMM 5-layer + NN penalty
DRMM 5-layer + KL penalty
DRMM 5-layer + KL and NN penalties

Test error (%) for a given number of labeled examples
NL = 50
NL = 100
NL = 1K
2.21 ± 1.36
21.73
22.10
2.46
0.91

3.33 ± 0.14
1.39 ± 0.28
2.12
1.32
1.06 ± 0.37
0.96
0.93 ± 0.065
13.41
12.28
1.36
0.57

2.40 ± 0.02
0.84 ± 0.08
2.35
2.26
0.71
0.6

Table 1: Test error for semi-supervised learning on MNIST using NU = 60K unlabeled images and NL ∈ {100, 600, 1K} labeled
images.

5.3. CIFAR10
We use CIFAR10 to test the semi-supervised learning performance of the DRMM on natural images. For
CIFAR10 training, we use the same 9-layer DRMM as
for SVHN. Stochastic gradient descent with exponentiallydecayed learning rate is still used to train the model. Table 4 presents the semi-supervised learning results for the
9-layer DRMM for NL = 4K, 8K images. Even though
we only use a simple SGD algorithm to train our model,
the DRMM achieves comparable results to state-of-the-art
methods (21.8% versus 20.40% test error when NL=4K
as with the Ladder Networks). For semi-supervised learning tasks on CIFAR10, the Improved GAN has the best
classification error (18.63% and 17.72% test errors when
NL ∈ {4K, 8K}). However, unlike the Ladder Networks
and the DRMM, GAN-based architectures have an entirely
different objective function, approximating the Nash equilibrium of a two-layer minimax game, and therefore, are not
directly comparable to our model.

5.4. Analyzing the DRMM using Synthetic Imagery
In order to better understand what the DRMM learns
during training and how latent variations are encoded in
the DRMM, we train DRMMs on our synthetic dataset
which has labels for important latent variations in the data
and analyze the trained model using linear decoding analysis. We show that the DRMM disentangles latent variations
over multiple layers and compare the results with traditional
DCNs.
Dataset and Training: The DRMM captures latent variations in the data [13, 14]. Given that the DRMM yields
very good semi-supervised learning performance on classification tasks, we would like to gain more insight into how
a trained DRMM stores knowledge of latent variations in

the data. To do such analysis requires the labels for the latent variations in the data. However, popular benchmarks
such as MNIST, SVHN, and CIFAR10 do not include that
information. In order to overcome this difficulty, we have
developed a Python API for Blender, an open-source computer graphics rendering software, that allows us to not only
generate images but also to have access to the values of the
latent variables used to generate the image.
The dataset we generate for the linear decoding analysis
in Section 5.4 mimics CIFAR10. The dataset contains 60K
gray-scale images in 10 classes of natural objects. Each
image is of size 32-by-32, and the classes are the same as
in CIFAR10. For each image, we also have labels for the
slant, tilt, x-location, y-location and depth of the object in
the image. Sample images from the dataset are given in
Figure 3.
For the training, we split the dataset into the training
and test set, each contains 50K and 10K images, respectively. We perform semi-supervised learning with NL ∈
{4K, 50K} labeled images and NU = 50K images without labels using a 9-layer DRMM with the same configuration as in the experiments with CIFAR10. We train the
equivalent DCN on the same dataset in a supervised setup
using the same number of labeled images. The test errors
are reported in Table 5.
Model
Conv Large 9-layer
DRMM 9-layer + KL and NN penalty

Test error (%)
NL = 4K NL =50K
23.44
6.48

2.63
2.22

Table 5: Test error for training on the synthetic dataset using
NU = 50K unlabeled images and NL ∈ {4K, 50K} labeled
images.

Figure 4: Linear decoding analysis using different numbers of
labeled data NL . The horizontal dashed line represents random
chance.

Figure 3: Samples from the synthetic dataset used in linear decoding analysis on DRMM and DCNs

Linear Decoding Analysis: We applied a linear decoding analysis on the DRMMs and the DCNs trained on the
synthetic dataset using NL ∈ {4K, 50K}. Particularly, for
a given image, we map its activations at each layer to the
latents variables by first quantizing the values of latent variables into 10 bins and then classifying the activations into
each bin using first ten principle components of the activations. We show the classification errors in Figure 4.
Like the DCNs, the DRMMs disentangle latent variations in the data. However, the DRMMs keeps the information about the latent variations across most of the layers
in the model and only drop those information when making
decision on the class labels. This behavior of the DRMMs
is because during semi-supervised learning, in addition to
object classification tasks, the DRMMs also need to minimize the reconstruction error, and the knowledge of the latent variation in the input images is needed for this second
task.

6. Conclusions
In this paper, we proposed a new approach for semisupervised learning with DCNs. Our algorithm builds
upon the DRMM, a recently developed probabilistic generative model underlying DCNs. We employed the EM

algorithm to develop the bottom-up and top-down inference in DRMM. We also apply variational inference and
utilize the non-negativity constraint in the DRMM to derive two new penalty terms, the KL and NN penalties,
for the training objective function. Our method achieves
state-of-the-art results in semi-supervised learning tasks on
MNIST and SVHN and yields comparable results to stateof-the-art methods on CIFAR10. We analyzed the trained
DRMM using our synthetic dataset and showed how latent
variations were disentangled across layers in the DRMM.
Taken together, our semi-supervised learning algorithm for
the DRMM is promising for wide range of applications in
which labels are hard to obtain, as well as for future research.

References
[1] Y. Bengio. Learning deep architectures for ai. Foundations
and Trends in Machine Learning, 2(1):1–127, 2009. 2
[2] C. M. Bishop. Pattern Recognition and Machine Learning,
volume 4. Springer New York, 2006. 3
[3] D. M. Blei, A. Kucukelbir, and J. D. McAuliffe. Variational inference: A review for statisticians. arXiv preprint
arXiv:1601.00670, 2016. 4
[4] I. Higgins, L. Matthey, X. Glorot, A. Pal, B. Uria, C. Blundell, S. Mohamed, and A. Lerchner. Early visual concept
learning with unsupervised deep learning. arXiv preprint
arXiv:1606.05579, 2016. 4, 5
[5] S. Ioffe and C. Szegedy. Batch normalization: Accelerating
deep network training by reducing internal covariate shift.
arXiv preprint arXiv:1502.03167, 2015. 10

[6] D. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. 4, 5
[7] D. P. Kingma, S. Mohamed, D. J. Rezende, and M. Welling.
Semi-supervised learning with deep generative models. In
Advances in Neural Information Processing Systems, pages
3581–3589, 2014. 2, 6, 9
[8] Y. Lecun, L. Bottou, G. B. Orr, and K.-R. Müller. Efficient
backprop. 1998. 10, 11
[9] D.-H. Lee. Pseudo-label: The simple and efficient semisupervised learning method for deep neural networks. In
Workshop on Challenges in Representation Learning, ICML,
volume 3, 2013. 2
[10] L. Maaløe, C. K. Sønderby, S. K. Sønderby, and
O. Winther. Auxiliary deep generative models. arXiv
preprint arXiv:1602.05473, 2016. 2, 6, 9
[11] A. Makhzani and B. J. Frey. Winner-take-all autoencoders.
In Advances in Neural Information Processing Systems,
pages 2773–2781, 2015. 2
[12] T. Miyato, S.-i. Maeda, M. Koyama, K. Nakae, and S. Ishii.
Distributional smoothing by virtual adversarial examples.
arXiv preprint arXiv:1507.00677, 2015. 6, 9
[13] A. B. Patel, T. Nguyen, and R. G. Baraniuk. A probabilistic
theory of deep learning. arXiv preprint arXiv:1504.00641,
2015. 1, 3, 6
[14] A. B. Patel, T. Nguyen, and R. G. Baraniuk. A probabilistic
framework for deep learning. NIPS, 2016. 1, 3, 4, 5, 6, 10
[15] A. Rasmus, M. Berglund, M. Honkala, H. Valpola, and
T. Raiko. Semi-supervised learning with ladder networks. In
Advances in Neural Information Processing Systems, pages
3532–3540, 2015. 2, 5, 6, 9, 10, 11
[16] S. Rifai, P. Vincent, X. Muller, X. Glorot, and Y. Bengio.
Contractive auto-encoders: Explicit invariance during feature extraction. In Proceedings of the 28th International
Conference on Machine Learning (ICML-11), pages 833–
840, 2011. 2
[17] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh,
S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein,
et al. Imagenet large scale visual recognition challenge.
International Journal of Computer Vision, 115(3):211–252,
2015. 5
[18] T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen. Improved techniques for training gans.
arXiv preprint arXiv:1606.03498, 2016. 6, 9
[19] J. T. Springenberg. Unsupervised and semi-supervised learning with categorical generative adversarial networks. arXiv
preprint arXiv:1511.06390, 2015. 6, 9
[20] J. Zhao, M. Mathieu, R. Goroshin, and Y. LeCun. Stacked
what-where autoencoders. arXiv preprint arXiv:1506.02351,
2016. 2

Model

Test error reduction (%)
NL = 50 NL = 100 NL = 1000

DRMM 5-layer + NN penalty
DRMM 5-layer + KL penalty
DRMM 5-layer + KL and NN penalties

−0.37
19.27
20.82

1.13
12.05
12.84

0.09
1.64
1.75

Table 2: The reduction in test error for semi-supervised learning on MNIST when using the KL penalty, the NN penalty, and both of them.
The trainings are for NU = 60K unlabeled images and NL ∈ {50, 100, 1K} labeled images

Model

Test error (%) for a given number of labeled examples
500
1000
2000

DGN [7]
Virtual Adversarial [12]
Auxiliary Deep Generative Model [10]
Skip Deep Generative Model [10]
ImprovedGAN [18]
DRMM 9-layer + KL penalty
DRMM 9-layer + KL and NN penalty

18.44 ± 4.8
11.11
9.85

36.02 ± 0.10
24.63
22.86
16.61 ± 0.24
8.11 ± 1.3
9.75
6.78

6.16 ± 0.58
8.44
6.50

Table 3: Test error for semi-supervised learning on SVHN using NU = 73, 257 unlabeled images and NL ∈ {500, 1K, 2K} labeled
images.

Model
Ladder network [15]
CatGAN [19]
ImprovedGAN [18]
DRMM 9-layer + KL penalty
DRMM 9-layer + KL and NN penalty

Test error (%)
4000
8000
20.40 ± 0.47
19.58 ± 0.46
18.63 ± 2.32
23.24
21.50

17.72 ± 1.82
20.95
17.16

Table 4: Test error for semi-supervised learning on CIFAR10 using NU = 50K unlabeled images and NL ∈ {4K, 8K} labeled images.

where a(`) ≡

Paper ID 2479

A. Model Architectures and Training Details
The details of model architectures and trainings in this
paper are provided in Table 6. The models are trained
using Stochastic Gradient Descent [8] with exponentiallydecayed learning rate. All convolutions are of stride one,
and poolings are non-overlapping. Full, half, and valid
convolutions follow the standards in Theano. Full convolution increases the image size, half convolution reserves
the image size, and valid convolution decreases the image
size. The mean and variance in batch normalizations [5]
are kept track during the training using exponential moving average and used in testing and validation. The implementation of the DRMM generation process can be found
in Section B. The set of labeled images is replicated until
its size is the same as the size of the unlabeled set (60K
for MNIST, 73,257 for SVHN, and 50K for CIFAR10). In
each training iteration, the same amounts of the labeled and
unlabeled images (half of the batch size) are sent into the
DRMM. The batch size used is 100. The values of hyperparameters provided in Table 6 are for NL = 100 in case
of MNIST, NL = 1000 in case of SVHN, and NL = 4000
in case of CIFAR10. NL is the number of labeled images
used in training. Also, DRMM of ConvSmall is the DRMM
whose E-step Bottom-Up is similar to the ConvSmall network [15], and DRMM of ConvLarge is the DRMM whose
E-step Bottom-Up is similar to the ConvLarge network [15].
Note that we only apply batch normalization after the convolutions, but not after the pooling layers.

B. Generation Process in the Deep Rendering
Mixture Model
As mentioned in Section 3 of the paper, generation in the
DRMM takes the form:
c(L) ∼ Cat({πc(L) })
g (`) ∼ Cat({πg(`) })
µcg ≡ Λg µc(L)
(1)

(2)

(L−1)

(L)

≡ Λg(1) Λg(2) . . . Λg(L−1) Λg(L) µc(L)
I ∼ N (µcg , σ 2 1D(0) ),
where ` ∈ [L] ≡ {1, 2, . . . , L} is the layer, c(L) is
the object category, g (`) are the latent (nuisance) vari(`)
(`+1)
(`)
are paramables at layer `, and Λg(`) ∈ RD ×D
eter dictionaries that contain templates at layer `. The
image I is generated by adding isotropic Gaussian noise
to a multiscale “rendered” template µcg . When applying the hard Expectation-Maximization (EM) algorithm,
we take the zero-noise limit. Here, g (`) = t(`) , a(`)



(`)

ax(`+1)


x(`+1) ∈X (`+1)

∈ RD

(`+1)

is a vec-

tor of binary switching variables
 that select the templates
(`+1)
(`)
(`)
to render and t
≡ tx(`+1) (`+1) (`+1) ∈ RD
∈X

x

is the vector of rendering positions. Note that x(`+1) ∈
X (`+1) ≡ {pixels in level ` + 1} (see Figure 2B) and
(`)
tx(`+1) ∈ {UL, UR, LL, LR} where UL, UR, LL and LR
stand for upper left, upper right, lower left and lower right
positions, respectively. As defined in [14], the intermediate
rendered image z (`) is given by:
(`)

z (`) ≡ Λg(`) z (`+1)

(11)

(`)

= Λt(`) ,a(`) z (`+1)
X
(`)
=
T (`)
t

x(`+1) ∈X (`+1)

(12)

x(`+1)



(`)
(`)
Zx(`+1) Γ(`) Ma(`)

(`+1)

x(`+1)

zx(`+1)
(13)

= DRMMLayer(z

(`+1)

(`)

,t

(`)

,a

,Γ

(`)

)

(14)


(`+1)
(`)
×D (`+1)
where Ma(`) ≡ diag a(`) ∈ RD
is a mask(`)

(`+1)

ing matrix, Γ(`) ∈ RF ×D
is the set of core templates
of size F (`) (without any zero-padding and translation) at
(`)
(`)
layer `, Z (`) ∈ RD ×F is a set of zero-padding opera(`)
(`)
(`)
tors, and Tt(`) ∈ RD ×D is a set of translation operators
(`)

to position t(`) . Elements of Z (`) and Tt(`) are indexed by
x(`+1) . Also, Γ(`) [:, x(`+1) ] are the same for x(`+1) in the
same channel of the intermediate rendered image z (`+1) .
Note that in the main paper, we call z (`) and z (`+1) intermediate rendered templates.
The DRMM layer can be implemented using convolutions of filters Γ(`) , or equivalently, deconvolutions of filters Γ(`)T . a(`) and t(`) are used to select rendering templates and positions to render, respectively. In the E-step
Top-Down Reconstruction, t̂(`) and â(`) estimated in the Estep Bottom-Up are used instead.

Model

Optimiser/Hyper-parameters

Dataset

DRMM
of ConvSmall
[15]

SGD [8]
learning rate 0.2 - 0.0001

MNIST

DRMM architecture
Input
E-step Bottom-Up

over 500 epochs
αH = 1, αRC = 0.2,
αKL = 1, αN N = 1
batch size = 100
Classes
E-step Top-Down
DRMM
of ConvLarge
[15]

SGD [8]
learning rate 0.2 - 0.0001

SVHN
CIFAR10

Input
E-step Bottom-Up

over 500 epochs
αH = 1, αRC = 0.5,
αKL = 0.2, αN N = 0.5
batch size = 100
Classes
E-step Top-Down

784 (flattened 28x28x1).
Conv 32x5x5 (Full),
Maxpool 2x2,
Conv 64x3x3 (Valid), 64x3x3 (Full),
Maxpool 2x2,
Conv 128x3x3 (Valid), 10x1x1 (Valid),
Meanpool 6x6,
Softmax.
BatchNorm after each Conv layer. ReLU activation.
10.
DRMM Top-Down Reconstruction. No BatchNorm.
Upsampling nearest-neighbor.
3072 (flattened 32x32x3).
Conv 96x3x3 (Half), 96x3x3 (Full), 96x3x3 (Full),
Maxpool 2x2,
Conv 192x3x3 (Valid), 192x3x3 (Full), 192x3x3 (Valid),
Maxpool 2x2,
Conv 192x3x3 (Valid), 192x1x1 (Valid), 10x1x1 (Valid),
Meanpool 6x6,
Softmax.
BatchNorm after each Conv layer. ReLU activation.
10.
DRMM Top-Down Reconstruction. No BatchNorm.
Upsampling nearest-neighbor.

Table 6: Details of the model architectures and trainings in the paper.

References
[1] Y. Bengio. Learning deep architectures for ai. Foundations
and Trends in Machine Learning, 2(1):1–127, 2009. 2
[2] C. M. Bishop. Pattern Recognition and Machine Learning,
volume 4. Springer New York, 2006. 3
[3] D. M. Blei, A. Kucukelbir, and J. D. McAuliffe. Variational inference: A review for statisticians. arXiv preprint
arXiv:1601.00670, 2016. 4
[4] I. Higgins, L. Matthey, X. Glorot, A. Pal, B. Uria, C. Blundell, S. Mohamed, and A. Lerchner. Early visual concept
learning with unsupervised deep learning. arXiv preprint
arXiv:1606.05579, 2016. 4, 5
[5] S. Ioffe and C. Szegedy. Batch normalization: Accelerating
deep network training by reducing internal covariate shift.
arXiv preprint arXiv:1502.03167, 2015. 10
[6] D. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. 4, 5
[7] D. P. Kingma, S. Mohamed, D. J. Rezende, and M. Welling.
Semi-supervised learning with deep generative models. In
Advances in Neural Information Processing Systems, pages
3581–3589, 2014. 2, 6, 9
[8] Y. Lecun, L. Bottou, G. B. Orr, and K.-R. Müller. Efficient
backprop. 1998. 10, 11
[9] D.-H. Lee. Pseudo-label: The simple and efficient semisupervised learning method for deep neural networks. In
Workshop on Challenges in Representation Learning, ICML,
volume 3, 2013. 2

[10] L. Maaløe, C. K. Sønderby, S. K. Sønderby, and
O. Winther. Auxiliary deep generative models. arXiv
preprint arXiv:1602.05473, 2016. 2, 6, 9
[11] A. Makhzani and B. J. Frey. Winner-take-all autoencoders.
In Advances in Neural Information Processing Systems,
pages 2773–2781, 2015. 2
[12] T. Miyato, S.-i. Maeda, M. Koyama, K. Nakae, and S. Ishii.
Distributional smoothing by virtual adversarial examples.
arXiv preprint arXiv:1507.00677, 2015. 6, 9
[13] A. B. Patel, T. Nguyen, and R. G. Baraniuk. A probabilistic
theory of deep learning. arXiv preprint arXiv:1504.00641,
2015. 1, 3, 6
[14] A. B. Patel, T. Nguyen, and R. G. Baraniuk. A probabilistic
framework for deep learning. NIPS, 2016. 1, 3, 4, 5, 6, 10
[15] A. Rasmus, M. Berglund, M. Honkala, H. Valpola, and
T. Raiko. Semi-supervised learning with ladder networks. In
Advances in Neural Information Processing Systems, pages
3532–3540, 2015. 2, 5, 6, 9, 10, 11
[16] S. Rifai, P. Vincent, X. Muller, X. Glorot, and Y. Bengio.
Contractive auto-encoders: Explicit invariance during feature extraction. In Proceedings of the 28th International
Conference on Machine Learning (ICML-11), pages 833–
840, 2011. 2
[17] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh,
S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein,
et al. Imagenet large scale visual recognition challenge.
International Journal of Computer Vision, 115(3):211–252,
2015. 5

[18] T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen. Improved techniques for training gans.
arXiv preprint arXiv:1606.03498, 2016. 6, 9
[19] J. T. Springenberg. Unsupervised and semi-supervised learning with categorical generative adversarial networks. arXiv
preprint arXiv:1511.06390, 2015. 6, 9
[20] J. Zhao, M. Mathieu, R. Goroshin, and Y. LeCun. Stacked
what-where autoencoders. arXiv preprint arXiv:1506.02351,
2016. 2

