Disentangled Variational Auto-Encoder for Semi-supervised Learning
Yang Li∗1 , Quan Pan1 , Suhang Wang3 , Haiyun Peng2 , Tao Yang 1 , and Erik Cambria2
1

arXiv:1709.05047v1 [cs.LG] 15 Sep 2017

2

School of Automation, Northwestern Polytechnical University
School of Computer Science and Engineering, Nanyang Technological University
3
Department of Computer Science and Engineering, Arizona State University

Abstract

for semi-supervised learning, the semi-supervised VAEs are typically composed of three main components: an encoder network qφ (z|x, y), a decoder pθ (x|y, z) and a classifier qφ (y|x).
In the application, the encoder, decoder and classifier can be
implemented using various models, e.g., MLP or CNN networks [19, 34]. Though the classifier plays a vital role in achieving the semi-supervised goal, it introduces extra parameters of
itself to learn. With the limited labeled data, it may not be an
optimal choice to introduce more parameters to VAE for semisupervised learning because it may memorize the limited data
with large quantities of parameters, namely overfiting.

In this paper, we develop a novel approach for semi-supervised
VAE without classifier. Specifically, we propose a new model
called SDVAE, which encodes the input data into disentangled
representation and non-interpretable representation, then the category information is directly utilized to regularize the disentangled representation via equation constraint. To further enhance
the feature learning ability of the proposed VAE, we incorporate
reinforcement learning to relieve the lack of data. The dynamic
framework is capable of dealing with both image and text data
with its corresponding encoder and decoder networks. Extensive Therefore, in this paper, we investigate if we can directly incorexperiments on image and text datasets demonstrate the effective- porate the limited label information to VAE without introducing
ness of the proposed framework.
a classifier so as to achieve the goal of semi-supervised learning and at the same time to reduce the number of parameters to
be learned. In particular, we investigate the following two challenges: (1) Without introducing classifier, how do we incorpo1 Introduction
rate the label information to VAE for semi-supervised learning?
and (2) How can we effectively use the label information for representation learning of VAE? In an attempt to solve these two
The abundant data generated online every day has greatly ad- challenges, we propose a novel semi-supervised learning model
vanced machine learning, data mining and computer vision com- named Semi-supervised Disentangled Variational Auto-Encoder
munities. However, manual labeling of large dataset is very time (SDVAE). SDVAE adopts the VAE with KKT conditions as it
and labor consuming. Sometimes it even requires domain knowl- has better representation learning ability than VAE. Unlike exedge. All the above results the majority of the data with limited isting semi-supervised VAEs that utilize classifiers, SDVAE enlabels. Therefore, semi-supervised learning, which utilizes both codes the input data into disentangled representation and nonlabeled and unlabeled data for model training, is attracting in- interpretable representation, and the category information is dicreasing attention [4, 22, 14, 33]. Existing semi-supervised mod- rectly utilized to regularize the disentangled representation as an
els can be generally categorized into those categories, i.e., dis- equation constraint. As the labeled data is limited, the labeled
criminative model, generative model, graph-based model and the information may not affects the model much. To remedy this,
combined model with those categories [29, 5, 9, 23, 31].
we further change the equation constraints into the reinforcement
Among various semi-supervised models proposed, the semi- learning format, which helps the objective gain the category insupervised generative models based on variational auto-encoder formation heuristics. The inverse auto-regression (IAF) is also
have shown strong performance in image classification [14, 19] applied to improve the latent variable learning. The proposed
and text classification [33]. The effectiveness of VAE for semi- framework is flexible in which it can deal with both image and
supervised learning comes from its efficiency in posterior dis- text data by choosing corresponding encoder and decoder nettribution estimation and its powerful ability in feature extract- works. The main contributions of the paper are:
ing from text data [2] and image data [14, 19]. To adapt VAE

• Propose a novel semi-supervised framework which directly
exploits the label information to regularize disentangled rep-

∗ liyangnpu@mail.nwpu.edu.cn

1

the RCE, namely maxθ,φ Eqφ (z|x) log pθ (x|z) [6]. Then the objective function is changed with the inequation constraints:

resentation with reinforcement learning;
• Extract the disentangle variable for classification and the
non-interpretable variable for the reconstruction from the
data directly; and

max Eqφ (zi |x) log pθ (x|z)
θ

(4)

subject to KL(qφ (z|x)||pθ (z) < )

• Conduct extensive experiments on image and text datasets Using KKT conditions [1], Eq.(4) can be rewritten as follows:
to demonstrate the effectiveness of the proposed SDVAE.
L̂(θ, φ; x, λ) = Eqφ (z|x) log pθ (x|z)
(5)
− λ(KL(qφ (z|x)||pθ (z)) − )

2

Preliminaries

where λ > 0 is the Lagrangian multiplier, which is used to penalize the deviation of the constraint KL(qφ (z|x)||pθ (z)) ≤ .
Given that λ > 0 and  > 0, we have

In this section, we introduce preliminaries that will be useful to
understand our model.

L̂(θ, φ; x, λ) ≥ Eqφ (z|x) log pθ (x|z) − λKL(qφ (z|x)||pθ (z))
(6)
If λ = 1, then Eq.(6) reduces to the original variational autoencoder problems that proposed by Kingma [16]. However, if
2.1 Variational Auto-Encoder
0 < λ < 1, then L̂(θ, φ, λ; x) > L(θ, φ, λ; x), which is closer
to the target log pθ (x). This is just the mathematical description
Variational Auto-Encoders (VAEs) have emerged as one of the
of the fact that the more information in the latent variable z, the
most popular deep generative models. One key step of VAE is to
tighter of the lower bound is. Through the KKT condition, a loose
evaluate pθ (x), which can be interpreted as
constraint over the decoder is introduced. Empirical results show
log pθ (x) = KL(qφ (z|x)||pθ (z|x)) + L(θ, φ; x)
(1) that VAE with KKT condition performs better than original VAE.
Thus, in this paper, we use VAE with KKT condition as our basic
where KL(Q||P ) is Kullback-Leibler divergence between two model.
distributions Q and P and L(θ, φ; x) is the evidence lower bound
(ELBO). It is defined as
L(θ, φ; x) = Eqφ (z|x) (− log qφ (z|x) + log pθ (x, z))

2.3

(2)

When there is label information y in the observed data, it is easy
to extend Eq.(6) to include label information as follows [14].

The term qφ (z|x) is to extract latent feature from the observed data x and it is called encoder generally. By minimizing KL divergence, we try to find qφ (z|x) that can approximate the true posterior distribution pθ (z|x). Because
L(θ, φ; x) is non-negative and log p(x) is fixed, then minimizing KL(qφ (z|x)||pθ (z|x) is equivalent to maximizing L(θ, φ; x).
We can rewrite L(θ, φ; x) as

L̂(θ, φ; x, y, λ) ≥ Eqφ (z|x,y) log pθ (x|z, y)
+ λ(log p(y) + log p(z) − log qφ (z|x, y))

(7)

To achieve the semi-supervised learning, [14] introduce a classifier qφ (y|x) to Eq.(7), which results in
X
U (θ, φ, y; x, λ) =
qφ (y|x)(L̂(θ, φ; x, y, λ)) + H(qφ (y|x))

L(θ, φ; x) = Eqφ (z|x) log pθ (x|z) − KL(qφ (z|x)||pθ (z)) (3)

y

where the first term in the RHS of Eq.(3) is the reconstruction
error (RCE), and the second term in the RHS is the KL divergence
between the prior and the posterior (KLD). Those two values play
different roles during the approximation. We will introduce them
in details in the next section.

2.2

Semi-supervised VAE

(8)
Apart from the Eq.(7) and Eq.(8), the classification loss over the
label information Ep(x,y) (log qφ (y|x)) is added into the objective
function when facing with the labeled data. However, in this paper, the discriminative information is added from scratch and an
equation constrained VAE is proposed, in order to highlight the
contribution of labeled data.

VAE with KKT Conditions

In practice, we find that the RCE is usually the main error, while 3 The Proposed Framework
the term of KLD is regarded as the regularization to enforce
(qφ (z|x)) to be close to pθ (z|x), which is relatively small. If
we constrain the KL divergence term into a small component  to In this section, we introduce the details of the proposed framegain a tighter lower bound, the goal is transformed to maximize work. Instead of using a classifier to incorporate the label infor2

3.2

mation, we seek to directly use label information to regularize the
latent representation so as to reduce the number of parameters.

SDVAE-I

The first way we consider is the cross entropy between y and v,
i.e.,
U = y log qφ (v|x)
(11)

3.1

Disentangled Representation

where y is the observed label information, qφ (·) is encoder for
the disentangle variable v. This is a popular loss function for
supervised learning and doesn’t introduce any new parameters.
Therefore, we choose this as the loss function for regularizing the
disentangled variable v. We name this method Semi-supervised
Disentangled VAE (SDVAE-I). By adding this loss function to
Eq.(10), the objective function of SDVAE-I is given as:

In order to incorporate the label information to the latent representation, we assume that the latent representation can be divided
into two parts, i.e., the disentangle variable and non-interpretable
variable. The disentangle variable captures the categorical information, which can be used for prediction task. Therefore, we
can use label information to constrain disentangled variable. The
(12)
L̂(θ, φ; x, λ, u) ≥ RE(z,v) − λ(KLz + KLv ) + µU
non-interpretable variable can be vectors comprised of any dimensions that combine other uncertain information from the data. µ is the weight parameter. When there is no labeled data, the
For the simplicity of notation, we use v to denote the disentan- equation condition will be U = 0.
gled variable and z to denote the non-interpretable representation. With v and z, the encoder can be rewritten as qφ (v, z|x).
We further assume that the disentangled variable and the non3.3 SDVAE-II
interpretable variable are independent condition on x, i.e.,

The drawback of the SDVAE-I is obviously, because the training
results depend on the number of the labeled data heavily. However, for semi-supervised learning, there is usually a small size
of the labeled data available. Thus, it is hard for the disentangle
It is a reasonable assumption, because given x, the categorical variable to capture the category information. To remedy this, we
information is only dependent on x and thus v, which captures the got inspired by the idea in [33]. The equation constraints can be
categorical information, is independent of z given x. This means expressed by the reinforcement learning, in which ELBO can be
there are seldom information about the category information in seen as the reward of the equation constraint. The disentangle
z, which is validated in the experiment part.
variable v acts as the agent which decides the output category
Now qφ (z|x) is the encoder for the non-interpretable representa- information. This can be seen in Eq.(14). Finally, a constant
tion, and qφ (v|x) is the encoder for the disentangle representa- number c is added to act as the bias. The parameter µ is changed
into as followed:
tion. Based on those assumptions, Eq.(7) is written as:
qφ (v, z|x) = qφ (v|x)qφ (z|x)

(9)

µ = RE(z,v) − (KLz + KLv ) + c
L̂(θ, φ; x, λ) ≥ Eqφ (z|x),qφ (v|x) log pθ (x|z, v)
+ λ(log p(v) + log p(z) − log qφ (z|x) − log qφ (v|x))

(13)

The partial update of this part is as followed:

(10)

∆φ = v(L + c)∇φ log qφ (v|x)

= RE(z,v) − λ(KLz + KLv )

(14)

Where L denotes the RE(z,v) − (KLz + KLv ).

where RE(z,v) = Eqφ (z|x),qφ (v|x) log pθ (x|z, v), which represents the reconstruction error given the variable (z, v). KLz and
KLv denote the KL(qφ (z|x)||p(z)) and KL(qφ (v|x)||p(v)) respectively. From the above equation, we can see that the categorical information is extracted from the data, i.e., captured in
disentangled variable v. Now if we have partial labels given, we
can directly use the label information to regularize v.

However, those terms only take effect over the labeled data. To
make up for this drawback, another term of log-likelihood expectation on disentangle variable v is added as the information entropy in Eq.(12), and this will be calculated both in labeled data
and the unlabeled data. It helps to reduce the large variance of the
disentangle information. Then the objective function in Eq.(12)
is changed into Eq.(15).

With v capturing the categorical information, there are many
ways to regularize v. Inspired by the work of [6], we add equation
L̂(θ, φ; x, λ, c) ≥ RE(z,v) − λ(KLz + KLv )
(15)
constraints on v over the ELBO, where the equation constraint is
+ y(β1 L + c) log qφ (v|x) + β2 H(qφ (v|x))
to enforce the disentangled representation v to be close to the label information y. In this work, we consider two ways to add the where y is the label information, β1 and β2 are the coefficient
constraint over the ELBO as discussed below.
parameters, and we name this model SDVAE-II.
3

3.4

With Inverse Autoregressive Flow

the same time are from both the labeled data and the unlabeled
data. Furthermore, we assume that these two variables are independent. However, it is not the same in the previous works, they
only extract the latent variable z from the data. When there is
no label information, label variable y infers from the x with the
shared parameters from qφ (z|x) or infers from z directly.

Because the two different latent variables are extracted from
the data directly, to make the posterior inference more flexible
and enhance the ability in disentangle representation in highdimension space, the inverse autoregressive flow (IAF) [35] is
applied in SDVAE-I and SDVAE-II. The chain is initialized with
the output µ0 and δ0 from the encoder. Together with the random sample ε ∼ N (0, I), the latent variable z is calculated as
z0 = µ0 + δ0 ⊗ ε. The way to update IAF chain is the same as
that in the LSTM shown in Eq.(16).

Then, based on different assumptions, there are differences with
the previous works in mathematics. The ELBO with two independent latent variable inferences is written as Eq.(10), and it is
different from that in Eq.(7) who only has one latent variable z
inference. Furthermore, if we ignore the assumption difference,
zt = δt ⊗ zt−1 + µt
(16) when facing with the labeled data in previous works, their objective function is a special case in Eq.(15) when β1 = β2 = 0.
where (δt , µt ) are the outputs of the auto-regression neural networks, whose input is the last latent variable zt−1 , and t is the When the label is missing, previous works apply the marginal
flow length.
posterior inference over the label information which is shown in

3.5

Eq.(8). In this paper, it is the inference for both latent variable
inference over the z and v, and this is shown in Eq.(17).

Training of SDVAE

U (x) = RE(z,v) − λ(KLz + KLv ) + β2 H(qφ (v|x))

(17)

The models can be trained end-to-end using mini-batch with the
ADAM optimizer [13]. The training algorithm is summarized in
Algorithm 1. In Line 1, we initialize the parameters. From Line
4 Experimental Results
3 to Line 5, we sample a mini-batch to encode the input data as
z and v. From Line 6 to Line 10, we apply IAF. We then update
the parameters from Line 11 to Line 13.
In this section, we conduct experiments to validate the effectiveness of the proposed framework. Specifically, we want to anAlgorithm 1 Training algorithm of the proposed models.
swer the following questions: (1) Is the disentangled representa1: Initialize the parameters , φ, θ
tion able to capture the categorical information? (2) Is the non2: repeat
interpretable variable helpful for the data reconstruction? (3) Is
3:
x ← Sample a miniBatch from the datapoints
the proposed framework effective for semi-supervised learning?
4:
 ← Random sample from the noise distribution
To answer the above questions, we conduct experiments on im5:
z, v ← qφ (z, v|x, )
age and text datasets, respectively.
6:
if IAF then
7:
for t < T do
8:
ẑ ← iaf (z, θ)
4.1 Experiments on Image Datasets
9:
end for
10:
end if
4.1.1 Datasets Description
11:
x̂ ← pθ (x|ẑ, v)
12:
g ← 5θ,φ L̂(θ, φ; x, λ, c)
Calculate the gradients of Eq.(15) for SDVAE-II, and Eq.(12) for For image domain, we choose two widely used benchSDVAE-I.
mark datasets for evaluating the effectiveness of SDVAE, i.e.,
13:
(θ, φ) ←Update with gradients g
MNIST [18] and SVHN [21]. In the MNIST, there are 55,000
14: until model convergence
data samples in the train set and 10,000 data samples in the test
set. In the SVHN, there are 73,257 data samples in the train set,
and 26,032 data samples in the test set. Both datasets contain 10
categories.
3.6 Discussion
The differences between the previous work [14, 33] and our work
4.1.2
will be discussed in this section.

Model Structure

Firstly, the assumptions are different. In this work, we assume For the image data, the encoder is a deep net composed of two
that the disentangle variable v and non-interpretable variable z at convolutional layers followed by two fully connected layers. The
4

Table 1: The classification errors on the MNIST data with part of labeled data
Models
NN
CNN
TSVM
CAE
MTC
AtlasRBF
Semi-VAE(M1)+TSVM
Semi-VAE(M2)
Semi-VAE(M1+M2)
SDVAE-I
SDVAE-I&IAF
SDVAE-II
SDVAE-II&IAF

([14])

600
11.44
7.68
6.16
6.3
5.13
5.72(±0.05)
4.94(±0.13)
2.59(±0.05)
2.75(±0.11)
2.74(±0.06)
2.49(±0.10)
1.97(±0.14)

ECVAE-I
convolutional layers are used to extract features from
the images
while the fully connected layers are used to convert the features to
the non-interpretable variable and the disentangle variable. The
decoder is a network composed of two fully connected layers to
map the latent features back to images. Dropout [26] is applied
to both the encoder and decoder networks.

4.1.3

1000
10.07
6.45
5.38
4.77
3.64
3.68(±0.12)
4.24(±0.07)
3.60(±0.56)
2.40(±0.02)
2.42(±0.08)
2.24(±0.08)
1.96(±0.09)
1.29(±0.11)

3000
6.04
3.35
3.45
3.22
2.57
3.49(±0.04)
3.92(±0.63)
2.18(±0.04)
1.70(±0.09)
1.33(±0.09)
1.58(±0.09)
1.00(±0.05)

SDVAE-I
ECVAE-I&IAF

SDVAE-I&IAF

SDVAE-II
ECVAE-II&IAF
5555555555 5

SDVAE-II&IAF
3333 3

00
0
888888888888
000000000 00
88888888888888888
6
6666 66666 00000
666 2 2222222 2
666666666666666666 666
88888888888888
6
0
6 2 2 22
666 6666
000 0
8
666 666666666 666666
0 00 0
6
6
8
0000000 0 00000 6 222222 2 222222222222222
6
8888888
6
8
4
666666666 66666 6
6666666666 6666 66 6
00 00
8 88 8 8
2
0
2
000 00 0000
0
2
2
0
2
6
6
6
6
2
4
2
2
0
0
6
0
2
2 22
8
0 0
6 66 66666666
4 6 6666666 6666666
4 4
888888 0000000 000
2 2222 2 2 2222
4444444 4444444444444444
6 666
000 00 0
6
8
4 444 44
2222 2 2 22 2
0
2 2222 222 2 2 22
44444444444444444444444 444
4
4
2
2
4
2
2
9
4
2
2
4444 444
4
4 44 444 44
11111111111
00000 00000
2
9
0 00000000000000 0000 0
4
55 8
1 611 1 111111
222222222222222222
1 111 1111
8 88888 88 8888 888
44 4 444
0 0 000 00
2 2222222 2222222221 2
00000000000 00000000000000
1111 1111 1
88 888888 8 888 88 888 5 555
4444 444444444444 44 4
33 3333 33333
1
2
7
2
2
2
2
2
7
3
0
8 8 8888 88888888 88 8
0
55
1 11 111 1
0 0000
32222 222222222222222
444 444 5 3 3 3 33333333 3
8
88888 8888
555 5 55
911 1 111111111111
2 2 27222
3 3 333 33 3333 33 3
8 55555555
8
3 33333333 3333333333 53 555 55 55 5555555
55 555 555
00
30 2 7 7777
3 33 3
55 555 5 5
9999 9
7
5
9
7
7
5
5
7
3
3
7 777 7
44444
9999999 999 9
99 9
3
33
7777777777777 77
444 44 4
4
4 4 4 999999999999999999999999999 9
555555555 55555 5
44 4 4 44
077777777777777
99999494 99 9 99999 9 9 99
55 5 5 5 555
3
11111
9999 9 9 9
4
9
9
55555555555555 555555555555555
39 7777777777777 9797
9
9 9999 9
1111111111111 99
44 4 9 99 99 9999999 9 99 9
7
7
5 55
7
7
1
7
7
7
77
9 9 9 99 9 9 9
11 1 111 1 1 5
55555 55 5
77 27
1 1
999949 9999999 9999 9 9999
7
99 9
1 11111111111111
77 777 7777 797 7 7
3
11111 111 1 1
3
3
3
3
3
3
777777 777 77777 777777777 7
1
3
3
1 111111 1
3 3333333 333 3
9
5
1 111 11 1
77777 77777777 77777777777777777
33333 33333333333 33333
11111111111 11
7 7 7 77 77 7 7 77
333 3333 333333333 3333333
1 111111 1
33 3

Disentangle Representation

ECVAE-II

35 55 5 555 55
5 3 55 5555555555555555
5 55555555555
9
5 55555555
3
3
3 3
5 565
33333333333333333 3593
6
6
33333 3333333 33333333
8
8
8888 8
333 3333 33
9
88888888888 8888
8
3 333333 333333
8
8
8
8
8
8
8
8
8
8
8
8
88 888 88 888 89
3 33333 3
5 66 6
9
8
8
8
8
8
8
6
4
66 6
8888888888888 8
3
9
444 444
666 666666 6
8 88888888
8 8
66666 6 66 6 6
44 44 44
8
44 44444444 444
6 6666 66 6
77
444444444444444444 4444 6 666666666 666 66666666
7
7
9
7
6 66666666666
444444444 444
77777777777 77
4
9 77777 7777 77777777777
1
6
111
9
7777777777777777777777
1
1
7
1
1
1 11
77777777 77 7
3
111111111 11111111111111
7
0000
777 772
111111111111
00 000000000000000 00009
8 32
0 00000 0 000 000000
111111
0000006 000000
1 111111111
22 22
00 0000000 00
2
1 11111111 1
02000000
2222 22222222 22 2 2
1 11
2222222222 2222222222222222222
2222 222222 2 22 2
22 4
2 222222 2222
22
99999 99999
99 9999
99999999999999999
9999999999999999999

The first experiment is to explore how the disentangle variable v
and non-interpretable variable z perform in the image reconstruction. The experiment is conducted on the MNIST dataset. In the
training data, we randomly select 3000 data samples as labeled
data and the remainings are unlabeled. The dimension of the disentangle variable v is 10 which is same as the category number,
and the dimension of z is 50.
We first train the model to learn the parameters. Then we use the
trained model to learn latent representation on the test data. After
learning the representations, we mask v and z in turn to see how
they affect the reconstruction of the input image. Two sample
results are shown in Fig.1. We also use t-SNE [28] to visualize v
of the testing data. The results from those four models (SDVAEI, SDVAE-I&IAF SDVAE-II and SDVAE-II&IAF) are shown in
Fig.2.

33333333333333
33333333333 333
33 33333333333 33333333 3
3 3333333
93
3

6666 666666666666
6 6666666666666666666666
6 6666666666 64
6666 66 66666
6 66
6
22 2
2 222 222222222 222
2222222222222222222 2222222 2
22222222 2222222 22222222
2 22222222222 24
2
2222

88888 8 8
88888888 8888888
88888888888888888888888
88
888888888888888
9 99
9 999999999
99 9999 9999
99 9999999999999999999999
9 999999 99999

11 11 11
11111111111111111111111111119
1111111111111 11111 1111111
1111 1111111111111
1

0
0000000000000000
00 00000000000000000
00000000000000000000
0
00 00 08
00

4 4444444 444 4
4 44 4 444
8 444444444444444444444444444
44444 4
4

7
77777777777777777777777
7777777 77777777
777777777777777777777777
777
7 777777777777

5 5
5555 55 5 55555
555 55555555555555555555 555555555
55555555555555
55555

Figure 2: The t-SNE distribution of the latent variable v from
proposed models, and different categories are in different colors
with number.

v mainly captures the categorical information, and it has little
influence over the reconstruction task. More specifically, from
Fig.2, we can see that images of the same class are clustered
Mask z
Mask v
together, implying that the disentangled representation captures
the categorical information. In addition, we find that cluster
SDVAE-I gives the worst visualization as clusters have intersections, while SDVAE-I&IAF and SDVAE-II&IAF give better
Figure 1: The first row in left figure and the right figure are the visualization, which suggests that SDVAE-I&IAF and SDVAEreconstruction images with the variable z and variable v masked II&IAF are better at capturing the categorical information.
respectively, and the images in the second row in both figures are
From Fig.1, we can see that when v is masked, z still reconstructs
the test images original.
the input image well, indicating that z is appropriate for reconstruction. To explore how variable z takes effect in the image
From Fig.1 and Fig.2, we can see that the disentangle variable reconstruction, we range a certain dimension of z from -2 to 2
5

4.2 Experiments on Text Dataset

on the specific labeled image, and the selected results are shown
in Fig.3. From the image, we can see that z can control differ3

4

3

0

4.2.1

Dataset Description

To test the model on text data, the IMDB data [20] is used. This
dataset contains 25,000 train samples and 25,000 test samples in
two categories.
Figure 3: The reconstruction images by varying z in a certain
dimension.
4.2.2 Model Structure
ent properties in image reconstruction with different dimensions,
such as italic controlling, bold controlling, transform control, and
In the application of the text data, the encoder is also the conthe style controlling, etc. These can be seen from images in Fig.3
volutional neural networks, but different from the case in image
left to right.
data, there are two convolutional neural networks which are referring from [12] parallelized together. One is extracting the feature
at the word level, and the other is extracting the feature at the
4.1.4 Semi-Supervised Learning
character level. As to the decoder, we applied the conditioned
LSTM [32], which is given as follows:
Furthermore, we conduct experiments to test the proposed models in semi-supervised learning on MNIST. We randomly select
ft = σ(Wf [z; v] + Uf ht−1 + bf )
x points from the training set as labeled data, where x is varit = σ(Wi [z; v] + Ui ht−1 + bi )
ied as {600, 1000, 3000}. The rest training data are used as unot = σ(Wo [z; v] + Uo ht−1 + bo )
labeled data. We compare with state-of-the-art supervised and
(18)
semi-supervised classification algorithms, which are used in [14].
It = Wc [z; v] + Uc ht−1 + bc
The experiments are conducted 10 times and the average accuct = ft ⊗ ct−1 + it ⊗ σ(It )
racy with standard deviation are showed in Table 1. Note that
ht = ot ⊗ relu(ct )
the performances of the compared methods are from [14] too.
From this table, we can see the proposed model SDVAE-II&IAF
performs best in classification and makes the least classification The conditional LSTM is same as the vanilla LSTM except for
errors (in black bold format) with small part of the labeled data. the current variable, which is replaced by the concatenation of
Although SDVAE-I performs not as good as other proposed mod- the latent variable z and v. The techniques of dropout [26] and
batch normalized [10] are both utilized in the encoder and deels, it still can achieve state-of-the-art results.
coder networks.
To further validate the observation, we also conduct the semisupervised learning over the SVHN, another popularly used
dataset. SVHN has 73,257 training samples and 26032 test samples. Among the training data, we randomly select 1000 data 4.2.3 Disentangle Representation
samples as labeled data and the rest as unlabeled data. The results
are shown in Table 2. Similarly, we can observe that SDVAE- We randomly select 20k samples from the training set as the laII&IAF gives the best performance.
beled data, and others are unlabeled during the training. Similarly, we use the t-SNE to visualize the disentangle variable
v ∈ N 2 and the non-interpretable variable z ∈ N 50 from the
proposed model on the test data and unlabeled data. Results are
showed in Fig.4,

Table 2: The results on the SVHN data
Method
KNN
TSVM
Semi-VAE(M1)+KNN
Semi-VAE(M1)+TSVM
Semi-VAE(M1+M2)
SDVAE-I
SDVAE-I&IAF
SDVAE-II
SDVAE-II&IAF

([14])

Test error rate
77.93% (±0.08)
66.55% (±0.10)
65.63% (±0.15)
54.33% (±0.11)
36.02% (±0.10)
47.32% (±0.13)
46.92% (±0.12)
44.16% (±0.14)
34.25% (±0.13)

From the left figure in Fig.4, we can see that the disentangle
representation v can clearly separate the positive and negative
samples while non-interpretable representation cannot, i.e., data
points from two clusters are interleaved with each other. This
suggests that the disentangle representation captures categorical
information well, and there is seldom categorical information in
non-interpretable variable.
6

Non-interpretable
variable z
1 01 0

1001
01

1
11 0
0 1
0
1
0 01
11 0
11001 1
11
1 0 010 010 1
0 10 101000 1 0
11111011
0 11
1
1 0 1 0 10 1
00 10
00 1
01 1
1100 01
1001
10
01
0
0 1 111 0 1000101 10
11010100
1 10100
11 1 01 0
011101
0101 00
0
0
1
0
101
0 1 1101011 1100010111100 10101
1
1
1111
000 111100
000
1011 10
0 110 10 0
01 01 01001001
0
11 1 1 1 01 1
0 00
0010100 110011101
1
011
001 1 11001110
100 100
01111
1 1 0 1 1 0 101111
11 001
101
01100
1
11 1 10 1001 01101001
0 10 1 0
10110111
10 0
011 1 00101101 0
1001 1011 00011
111
0 110000
1
000001011000
0 10 0 110110101 00
000 00101111 011
1
011
101 00 000 011 00000 1
1 10
00
10 11
000110 10100110
0 0 1110111010 1
1 101 1111010
1
1 10 10111
11
00001 100000
0
1
0
0
0
0
1
0
0
0
1
0
0
1
0
0
0
0
0
0100 0 1 11011 0010 1 0 1101 0 10001 0
1 11
10 00 11
100
00
0 11 0
10 000 10 001
01 1 011 1
101111 1 0
01
0 100 0
0
0 01 100
111011101010
1100
1 01 1
0 0111
0110
1 1
110 1110
10 001 111 10 01011000
0 0
00001110 00 0 0101
10 1
0 1
1 0010 0 01
1 0010 00 1
00
011 0
0 01 11 1 00 0 1011
0
0 0 1 11 100111000 101 101
0 1011010111 1 101 10 101
0 1 1 100 1
1
0
1
1
0
0
00 1
10 0 1 000100101
1 1 00
1 0 0 0 1011 01 1 1 1 11100 0
11 100011
1 0
11 1 1 01 010 1001 1 1100 0 1001 0
101 0
1 101010 0
0 1
000 10 00 0101 110 0
0 1
10 0
1
011
0 1
0 1 010 1 101 00 1 1 1 01
1
1
1
1
1
0
0
1 0 00
0
1
11
0
1
1 01 1 0
0
1
0 0
0

1

4.3

vd
Disentangle variable
0 00

00
0 000 00 0 00
0
00 0
1
1 111111
00000 000000 0
1111 1 11
0 000
000 0 000
0
11 1 1 11
11 1111
1
00 0 0
0
111111111
111 111 11 111
00000000
0000 00 000
1
0
0
111111 1
0 0000 0
111 111
0
0
0
1
0
1
1
1
000000 0
0
1 1 1
0000
0
0000
000
0
0
1 1 11 111
111
10
00
000 00
0
1
0 0 0000 0 00
1 11 1 1 11 1111
000 000 000000
00
1 1 1111 11111
0 0 00
0 0 00
00
1111 1 11 1 11
0
0
1
0
0
0
0
0
1
1
1
1
0
0
1
00 0
0
1
0 0000
11
11
11
1 1 111111 1
00 0 000000 00
1 11
1
000 0
111
111
0
1
1
1
11 1111 1
00 0 0
00
1
11
1 111 111
1111
00
1111111
1 1111 1
1
11
0
1111 1 1 11 1 111 1
00
11 11
00
11
1 11
00
0
0
1
00
1 11
0
1 11111
111
1 1111
0
000 00 0
1
0 0
111 1111
1 11 11
0
11
00 000 000
000000
0 00
1111
1
111 11111 1111
0
0
0
0
0
11 1 11111 11
0
1
1
1
0
1
0
1
0 00 00 000 0 000 0
1 1 1111 1 1111 1 111 11
111
0
0 0 0 00
1
1
1
0
0
1
1
1
0 0 0000
11 1 1 1 1
0 00
00
1 1
1 11
00000 0000 00
00000
00
11
00 0 0 0
1111 1 1 1
00 0
00
0 0 0000
1
0
11 11
0
0
1
1
1
0 0 0
00 00 000 0 00
111
1 1 11 1 11
0 0
0
00 0
00 000 00
11
1 1 1 111111
0000
0
000000 0
111
1111111
1
1111
0
000 0 00
11
0
0
1
0
0
1
0
0
1
1
0
1 11
00 00
1 1 11 1
0
11 111
11
1111
00 000000
11 1 111111
0 0000 00
0
0
0 000
1111 11 1
0 0000
11111
00 0
0
0
0
0000
0000
1111
000
00 0000000
111 11 1
11 11 1111
00000
11 111
00 000
00
11
11
000 0
11
1
0 00

Parameters Analysis

There are several important parameters need to be tuned for the
model, i.e., λ, β1 , β2 and the length of IAF. In this section, we
conduct experiments to analyze the sensitiveness of the model to
the parameters.

4.3.1

Effects of λ and the IAF Length

(a) Unlabeled Data
Non-interpretable
variable z
01

0 11
01 1
1 11 1 0 0
0 0 0 0 101 1
000
0 10 10000 1 10 011 00 00 11 11 11 1
1 1
1 0
1
1
0 11
10
0 001 00101 10 0 0 0
00001 00 1 01 1 0
0000 0 11101 11 0 0 0 01
01 1 01 00 0 00 1 0
0
1
0
0
11 1 10 00
1
1
0
1110 1 10
11
01
1
0 0 010 10 0 111 0 110 1 10 1
0 1 0011110 1110 010 1011011010001 111 0 001001 01 10 111
0 0 0001
1
0
0
0
1
1
1
1
0
1
0
0 0
1
11 0
1
01
1111 00 1 00 100101 00111 10100
00 0 101 1 00 0
01
10 001 0
1
1 1
01
01111 10
1101 01 0001 1 001 0
1
010101110
0
0 10000 0
00 100100
0
1
0
0 1 1
1
1
1
1
0
1
1
1
0 0
1 00000 0000
10
11110 00 0010
11
101 1 00 0
10
0 0
1
1 000 0 1 001 0
1 0 0010 1100 11111 1010000 1111
0000 010001
0 1 0
0
11 0 0
1 0 0 0 000 1 0
11
1 10010 001 1101
1
10
0010 1 11111
1
00 0 110100
1000 1101001
10 00 11111 01001 11 0 0
10 1 00
1 1 11
0
1
0
0
1
0
0
1
1
1 01 111 0001 1 0 1 0111 101001 1 001 1
0 0 00 0 1 10 11001
0
0
0 0
1
01
0 1 11101001
01 000 1001 10 0100
0 0011 01 101
00 00 0010 1
10 001 1
0
0 001010110
00 10
110010 1011 0010 1 1 0
0011 0
1110 100
011 10100101100 01 0 00100 011 011 1 1
0 00
1 000 10 010 1
0
1
0
1
0 0 0
0
0 0 0110 1
0
1
0
1
0 01 010 010110 01 1101 10
10 0 11 0
1
00 1 110 110 00 11 0 0001
001
0
101 0
1
0 0 00111 0 0010 0100 11 01101
1 0111 10000 111 00 010 110 1 0
10 0
11
1 0
0 1 11
0
11 111 00000 0 000 1101 00 11 0 0
1 00
10
1
1
0 1 00
0 00
1
0
1
1
1
0 0 01 0 1 1 01 1 0 11
1 1
1 0
0 0 1 0
000 1 1
01011 01 1 1 1
1
0
1
000
1
0
0
10 0 000 0 0 1
0
1

Disentangle0 0 variable vd

We firstly evaluate λ and the length of the IAF chain which are
proposed in the works of β-VAE [6] and IAF [15]. These experiments are conducted over the MNIST training dataset.

00
1
1
000
010
11
1
1
1
0 1101 00
010 10001
11 1
0 0
0 10
100
10
110
1
1
1
00
1010 0 00001
0100
11111 101111
1
1
0
00
0 0 01
1000
1
1
1
000000
01011
0 000 0
01 1 10101
11
11
1
1100 00 00000 0
010
001
1
11111
000
1
000
1
0
1
1
1
0
101 1 1
111
0 00
11
1 00
1
00000
1
1
00 1
1 10
00
1111
1111 111111
11 1
10010
00 0 00000
1
1 0110
1111
1
1
1
0
0 000
0
1
1
0
1
1
1
1
11111 111
0
10 1 0 01 0000 00 00001
0 011
1
0
0000 0 00
11 11111
110 0 0 1 0 1
0
1
1
0
1
0
1
0
0
000 000 00 0000
11 1 1 1
1 1
10
101 00000
00 00000
11 1111
0
10 010
00
000000
000 10000
0
01111111
10
1 1 11
0000
0 000000
10
0 10
1111 1
11 1
0 00000
00000000
000000 00000000
0000
00
1111 0 00
1 0101
1 00
0 000
1
0
11 1111
0
000 000 00
0
1
1
0
1
0
1
1
1
0
0
1
1
1
0
1
0
0
0
1111111
0 0 000 0 0000 0 000
1 1 00
0
11 1 11
1
0
0
11111
1
11 0 0 00101 000 0000 00
0 11111
1
11111
111011
01 1 1
0
0
0
0
111
111 111
0
0
001
1111
0
0000 0
1
11
1 1
0
0 100
0 0
00
1
11
1
0
101111111 1 0 0
11
1
00
11111011011
00
00
10
1 1 000
010 0100
01100
001000
11 1111
11110
0
11 01
00
1 00
0
00 0001
1111
1
0
00 000
10 0
1
00
000
10
11
1 10
10
0
11
0 00 00
0
00
11010
0 0000
1 000
0
11
1101
110
000000
111
001
0
1
1
0
0
1
0
0
1
11
0
1
00
0
111
0
11
10
1
1
11
11

The objective function in the proper λ finding is depicted in
Eq.(6). Results with different λ values are shown in Fig.5(a)
From the results, we can see that it is better for λ to have a small
value, which not only leads to a rich information in the latent
variable but also gets a better reconstruction error. But as de(b) Test Data
scribed before, the large value of KL-divergence is also the cause
Figure 4: The left figure is the t-SNE distribution of the non- of overfitting or the underfitting for the model. However, in the
interpretable variable z, the right figure is the t-SNE distribution case of λ = 0.1, there is a low reconstruction error, which is the
of the disentangle variable v correspondingly. Different cate- sign of the good performance.
gories are in different colors with number.
Then the model structure about the IAF chain is built according
to the Eq.(16), and the results with different length are shown in
4.2.4 Semi-Supervised Learning
the right figure in Fig.5(b). From the figure, we can see that it is
not good to set the chain too long if it is a long IAF. The RCEs are
We further conduct semi-supervised classification on the text not so good together with the KLDs, and the latent variable isvery
dataset using the representation learned from previous experi- unstable. On the contrary, there is a stable increase about the
ments and fine tuning the model. Similarly, we compare with KL divergence, and a stable decrease reconstruction error when
state-of-the-art semi-supervised learning algorithms. The aver- the length of the IAF chain is set to 1. This means that under
age test error rate is reported in Table 3. From the results, we can the good reconstruction, the latent variable captures more useful
see that: (i) SDVAE-II&IAF outperforms the compared meth- information. This is also validated in the results of the SDVAEods, which implies the effectiveness of the proposed framework I&IAF and SDVAE-II&IAF. Thus, in the experiments about the
for semi-supervised learning; and (ii) As we add reinforcement IAF, its length is set to 1 by default.
Powered by TCPDF (www.tcpdf.org)

Powered by TCPDF (www.tcpdf.org)

learning and IAF, the performance increases, which suggests the
two components contribute to the model.

4.3.2

Table 3: The results on the IMDB data
Method
LSTM ([4])
Full+Unlabeled+BoW ([20])
WRRBM+BoW ([20])
NBSVM-bi ([30])
seq2-bown-CNN ([11])
Paragraph Vectors ([17])
LM-LSTM ([4])
SA-LSTM ([4])
SSVAE-II&LM ([33])
SDVAE-I
SDVAE-I&IAF
SDVAE-II
SDVAE-II&IAF

Effects of β1 and β2

To decide the parameter β1 and β2 that in SDVAE-II, we made
the grid search both on the text data and the image data. For the
image data, the experiment is conducted on the SVHN dataset
with 1000 labeled samples. Experimental result with β1 ranges
from 0.1 to 1000, and β2 ranges from 0.01 to 100 are shown in
Fig.6(a). For the text data, the experiment is conducted on the
IMDB data with 20,000 labeled samples. Experimental result
with β1 and β2 range from 0.1 to 1000 are shown in Fig.6(b).

Test error rate
13.50%
11.11%
10.77%
8.78%
7.67%
7.42%
7.64%
7.24%
7.23%
12.56%
11.60%
7.37%
7.18%

From the Fig.6(a), we can see that, an acceptable range for β1 in
the image data is [0.1:100] and [0.01:10] for the β2 . Especially,
when β1 = 0.1 and β2 = 1, it is achieving the best result.
For the text data, the results in the Fig.6(b) show that the accuracy
7

λ=0.1
λ= 0 . 5
λ= 1 . 0
λ= 2 . 0

1 5 0
1 4 0

λ=
λ=
λ=
λ=

0 .1
0 .5
1 .0
2 .0

5 0
4 0

1 3 0

K L D

R C E

3 0
1 2 0
2 0
1 1 0

(a) The Image data

1 0

1 0 0
9 0

Figure 6: The grid search results for the proper β1 and β2 finding.
0

0

3 0

6 0

9 0

1 2 0

1 5 0

E p o c h

semi-supervised VAE all use a parametric classifier, which increases the burden to learn more parameters given the limited
labeled data. The proposed framework incorporates the label information directly into the disentangled representation and thus
avoids the parametric classifier.

(a) Validate λ
t=1

t=1

1 5 0

1 3 0

1 8

t=1 0
t=2 0
t=3 0

t=1 0
t=2 0
t=3 0

1 4 0

1 6
1 4
1 2

1 1 0

1 0

1 0 0

8

9 0

6

8 0

4

7 0

2

6 0

Variants of VAE Because of the great potential of VAE in image and text mining, various models based on VAE are proposed
to further improve its performance [16, 6, 7, 15]. For example,
[6] apply the KKT condition in the VAE, which gave a tighter
lower bound. Similarly, [3] introduce importance weighting to
VAE, which also tries to give a tighter bound. [24] consider the
stein based sampling to minimize the KL divergence. [7] rewrite
the evidence lower bound objective by decomposition, and give
a clear explanation for each term. To extend the flexible of the
posterior inference, IAF is introduced [15] which improves the
VAE a lot.

K L D

R C E

1 2 0

0
0

2 0

4 0

6 0

8 0

1 0 0

1 2 0

1 4 0

E p o c h
(b) Validate Length of IAF

Figure 5: The left y-axis in each figure is the reconstruction error
(BCE) which is axis of the solid lines, and the right y-axis is the
KL divergence (KLD) which is axis of the dash lines.

6

is not sensitive to β2 . However, when β1 is small, the result will
be more precise. In conclusion, it is better to set β1 to 0.1 and β2
can be set randomly.

5

(b) The Text data

Conclusions

In this work, we propose models that extract the disentangle variable v and the non-interpretable variable z from data at the same
time. The disentangle variable is designed to capture the category information and thus relieves the use of classifiers in semisupervised learning. The non-interpretable variable is designed to
reconstruct the data. Experiments show that it could even reflect
certain textual features, such as italic, bold, transform and style
in the hand writing digital data during the reconstruction. These
two variables cooperate well and each performs its own functions
in the SDVAE. The IAF improves the model effectively on the
basis of SDVAE-I and SDVAE-II. Especially in which, SDVAEII&IAF achieves the state-of-the-art results both in image data
and the text data in the semi-supervised learning tasks.

Related Works

Semi-supervised VAE Semi-supervised learning is attracting increasing attention, and lots of works are proposed [33, 8, 22, 31,
23, 14, 27, 5]. Those works can be divided into the discriminative models [29, 4], the generative models [33, 8, 22], graph based
models [27], and the combined model with those [5]. Because of
the effectiveness of deep generative models in capturing the data
distribution, semi-supervised models based on deep generative
models such as generative adversarial network [25] and variational auto-encoder (VAE) [14] become popular. Semi-VAE [14]
incorporates the learned latent variable into the classifier and improves the performance greatly. SSVAE [33] extends Semi-VAE
for sequence data, and also demonstrates its effectiveness in the
semi-supervised learning on the text data. The aforementioned

References
[1] Dimitri P Bertsekas. Nonlinear programming. Athena scientific Belmont, 1999.
8

[2] Samuel R Bowman, Luke Vilnis, Oriol Vinyals, and et al. [16] Diederik P Kingma and Max Welling. Auto-encoding variGenerating sentences from a continuous space. arXiv
ational bayes. arXiv preprint arXiv:1312.6114, 2013.
preprint arXiv:1511.06349, 2015.
[17] Quoc Le and Tomas Mikolov. Distributed representations
[3] Yuri Burda, Roger Grosse, and Ruslan Salakhutdiof sentences and documents. In Proceedings of the ICML,
nov. Importance weighted autoencoders. arXiv preprint
pages 1188–1196, 2014.
arXiv:1509.00519, 2015.
[18] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick
[4] Andrew M Dai and Quoc V Le. Semi-supervised sequence
Haffner. Gradient-based learning applied to document
learning. In Proceedings of the NIPS, pages 3079–3087,
recognition. Proceedings of the IEEE, 86:2278–2324, 1998.
2015.
[19] Lars Maaløe, Casper Kaae Sønderby, Søren Kaae Sønderby,
[5] Jingrui He, Jaime G Carbonell, and Yan Liu. Graph-based
and Ole Winther. Auxiliary deep generative models. arXiv
semi-supervised learning as a generative model. In Propreprint arXiv:1602.05473, 2016.
ceedings of the IJCAI, volume 7, pages 2492–2497, 2007.
[20] Andrew L Maas, Raymond E Daly, Peter T Pham, and et al.
[6] Irina Higgins, Loic Matthey, Arka Pal, and et al. beta-vae:
Learning word vectors for sentiment analysis. In ProceedLearning basic visual concepts with a constrained variaings of the ACL: Human Language Technologies-Volume 1,
tional framework. In Proceedings of the ICLR, 2017.
pages 142–150, 2011.
[7] Matthew D Hoffman and Matthew J Johnson. Elbo surgery: [21] Yuval Netzer, Tao Wang, Adam Coates, and et al. Reading
yet another way to carve up the variational evidence lower
digits in natural images with unsupervised feature learning.
bound. In Proceedings of the NIPS, Workshop in Advances
In Proceedings of the NIPS, Workshop on deep learning and
in Approximate Bayesian Inference, 2016.
unsupervised feature learning, volume 2011, page 5, 2011.
[8] Zhiting Hu, Zichao Yang, Xiaodan Liang, and et al. Toward
[22] Augustus Odena. Semi-supervised learning with generative
controlled generation of text. In Proceedings of the ICML,
adversarial networks. arXiv preprint arXiv:1606.01583,
pages 1587–1596, 2017.
2016.
[9] Gao Huang, Shiji Song, Jatinder ND Gupta, and Cheng
[23] Yong Peng, Bao-Liang Lu, and Suhang Wang. Enhanced
Wu. Semi-supervised and unsupervised extreme learning
low-rank representation via sparse manifold adaption for
machines. IEEE transactions on cybernetics, 44(12):2405–
semi-supervised learning. Neural Networks, 65:1–17, 2015.
2417, 2014.
[10] Sergey Ioffe and Christian Szegedy. Batch normalization: [24] Yunchen Pu, Zhe Gan, Ricardo Henao, and et al. Stein
variational autoencoder. arXiv preprint arXiv:1704.05155,
Accelerating deep network training by reducing internal co2017.
variate shift. In Proceedings of the ICML, pages 448–456,
2015.
[25] Jost Tobias Springenberg.
Unsupervised and semisupervised
learning
with
categorical
generative adversarial
[11] Rie Johnson and Tong Zhang. Effective use of word order
networks.
arXiv
preprint
arXiv:1511.06390,
2015.
for text categorization with convolutional neural networks.
arXiv preprint arXiv:1412.1058, 2014.

[12]
[13]

[14]

[15]

[26] Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, and
et al. Dropout: a simple way to prevent neural networks
Yoon Kim. Convolutional neural networks for sentence
from overfitting. Journal of Machine Learning Research,
classification. arXiv preprint arXiv:1408.5882, 2014.
15(1):1929–1958, 2014.
Diederik Kingma and Jimmy Ba. Adam: A method for
stochastic optimization. arXiv preprint arXiv:1412.6980, [27] Amarnag Subramanya and Partha Pratim Talukdar. Graphbased semi-supervised learning. Synthesis Lectures on
2014.
Artificial Intelligence and Machine Learning, 8(4):1–125,
Diederik P Kingma, Shakir Mohamed, Danilo Jimenez
2014.
Rezende, and Max Welling. Semi-supervised learning with
deep generative models. In Proceedings of the NIPS, pages [28] Laurens Van der Maaten and Geoffrey Hinton. Visualizing
data using t-sne. Journal of Machine Learning Research,
3581–3589, 2014.
9:2579–2605, 2008.
Diederik P Kingma, Tim Salimans, Rafal Jozefowicz, and
et al. Improved variational inference with inverse autore- [29] V Vapnik and A Sterin. On structural risk minimization or
gressive flow. In Proceedings of the NIPS, pages 4743–
overall risk in a problem of pattern recognition. Automation
4751, 2016.
and Remote Control, 10(3):1495–1503, 1977.
9

[30] Sida Wang and Christopher D Manning. Baselines and bigrams: Simple, good sentiment and topic classification. In
Proceedings of the ACL: Short Papers-Volume 2, pages 90–
94, 2012.
[31] Suhang Wang, Jiliang Tang, Charu Aggarwal, and Huan
Liu. Linked document embedding for classification. In Proceedings of the 25th ACM International on Conference on
Information and Knowledge Management, pages 115–124.
ACM, 2016.
[32] Tsung-Hsien Wen, Milica Gasic, Nikola Mrksic, and et al.
Semantically conditioned lstm-based natural language generation for spoken dialogue systems.
arXiv preprint
arXiv:1508.01745, 2015.
[33] Weidi Xu, Haoze Sun, Chao Deng, and Ying Tan. Variational autoencoder for semi-supervised text classification.
In Proceedings of the AAAI, pages 3358–3364, 2017.
[34] Xinchen Yan, Jimei Yang, Kihyuk Sohn, and Honglak Lee.
Attribute2image: Conditional image generation from visual
attributes. In Proceedings of the ECCV, pages 776–791,
2016.
[35] Zichao Yang, Zhiting Hu, Ruslan Salakhutdinov, and Taylor
Berg-Kirkpatrick. Improved variational autoencoders for
text modeling using dilated convolutions. arXiv preprint
arXiv:1702.08139, 2017.

10

