Improved Variational Autoencoders for Text Modeling using Dilated Convolutions
Zichao Yang Zhiting Hu Ruslan Salakhutdinov Taylor Berg-Kirkpatrick


[
Improved Variational Autoencoders for Text Modeling using Dilated Convolutions
















Zichao Yangcmu
Zhiting Hucmu
Ruslan Salakhutdinovcmu
Taylor Berg-Kirkpatrickcmu


cmuCarnegie Mellon University

Zichao Yangzichaoy@cs.cmu.edu







boring formatting information, machine learning, ICML

0.3in
]









  




Recent work on generative text modeling has found that variational
autoencoders (VAE) with LSTM decoders perform worse than
simpler LSTM language models <cit.>. This negative
result is so far poorly understood, but has been attributed to the
propensity of LSTM decoders to ignore conditioning information from the encoder. In this paper, we experiment with a new type of decoder for VAE: a dilated CNN.
By changing the decoder's dilation architecture, we control the size of context
from previously generated words. In experiments, we find that there is a trade-off
between contextual capacity of the decoder and effective use of encoding information. We show that when carefully managed, VAEs can outperform LSTM language models. We demonstrate perplexity gains on two datasets, representing the first positive language modeling result with VAE.

Further, we conduct an in-depth investigation of the use of VAE
(with our new decoding architecture) for semi-supervised and unsupervised
labeling tasks, demonstrating gains over several strong baselines.








§ INTRODUCTION

Generative models play an important role in NLP, both in their use as language models and because of their ability to effectively learn from unlabeled data.
By parameterzing generative models using neural nets, recent work has proposed model
classes that are particularly expressive and can pontentially model a wide range of phenomena in language and other modalities. We focus on a specific
instance of this class: the variational autoencoder[The name VAE is often used to refer to both a model class and an associated inference procedure.] (VAE) <cit.>.








The generative story behind the VAE (to be described in detail in the next section)
is simple: First, a continuous latent representation is sampled from a multivariate Gaussian. Then, an output is sampled from a distribution parameterized by a neural decoder, conditioned on the
latent representation. The latent representation (treated as a latent variable during training) is intended to give the model more expressive capacity when compared with simpler
neural generative models–for example, conditional language models. The choice of decoding architecture and final output distribution, which connect the latent representation to output, depends on the kind of data being modeled. The VAE owes its name to an accompanying variational technique  <cit.> that has been successfully used to train such models on image data  <cit.>.


The application of VAEs to text data has been far less successful <cit.>. The obvious choice for decoding architecture for a textual VAE is an LSTM, a typical workhorse in NLP. However, <cit.> found that using an LSTM-VAE for text modeling yields higher perplexity on held-out data than using an LSTM language model. In particular, they observe that the LSTM decoder in VAE does not make effective use of the latent representation during training and, as a result, VAE collapses into a simple language model.

























Related work <cit.> has used simpler decoders that model text as a bag of words. Their results indicate better use of latent representations, but their decoders cannot effectively model longer-range dependencies in text and thus underperform in terms of final perplexity.

Motivated by these observations, we hypothesize that the contextual capacity of the decoder plays an important role in whether VAEs effectively condition on the latent representation when trained on text data. We propose the use of a dilated CNN as a decoder in VAE, inspired by the recent
success of using CNNs for audio, image and language modeling
 <cit.>. In contrast with prior work where extremely large CNNs are used, we exploit the dilated CNN for its flexibility in varying the amount of conditioning context.
In the two extremes, depending on the choice of dilation, the CNN decoder can reproduce a simple MLP using a bags of words representation of text, or can reproduce the long-range dependence of recurrent architectures (like an LSTM) by conditioning on the entire history. Thus, by choosing a dilated CNN as the decoder, we are able to conduct experiments where we vary contextual capacity, finding a sweet spot where the decoder can accurately model text but does not yet overpower the latent representation.

We demonstrate that when this trade-off is correctly managed, textual VAEs can perform substantially better than simple LSTM language models, a finding consistent with recent image modeling experiments
using variational lossy autoencoders <cit.>.
We go on to show that VAEs with carefully selected CNN decoders can be quite effective for semi-supervised classification and unsupervised clustering,
outperforming several strong baselines (from  <cit.>) on both text categorization and sentiment analysis.

Our contributions are as follows: First, we propose the use of a dilated CNN as a new decoder
for VAE. We then empirically evaluate several dilation architectures with different capacities, finding that
reduced contextual capacity leads to stronger reliance on latent representations.
By picking a decoder with suitable contextual capacity, we find our VAE performs better than
LSTM language models on two data sets.




We also explore the use of dilated CNN VAEs for semi-supervised
classification and find they perform better than strong baselines from
 <cit.>. Finally, we verify that the same framework can be used
effectively for unsupervised clustering.




§ MODEL

In this section, we begin by providing background on the use of variational autoencoders for language modeling.
Then we introduce the dilated CNN architecture that we will use as a new decoder for VAE in experiments.
Finally, we describe the generalization of VAE that we will use to conduct experiments on semi-supervised classification.




 §.§ Background on Variational Autoencoders

Neural language models <cit.> typically generate each token
x_t conditioned on the entire history of previously generated tokens:

    p(𝐱) = ∏_tp(x_t | x_1, x_2, ..., x_t-1).

State-of-the-art language models often parametrize these conditional probabilities
using RNNs, which compute an evolving hidden state
over the text which is used to predict each x_t. This approach, though effective in modeling text, does not explicitly model variance in higher-level properties of entire utterances (e.g. topic or style) and thus can have difficulty with heterogeneous datasets.

<cit.> propose a different approach to generative text modeling inspired by related work on vision <cit.>. Instead of directly modeling the joint probability p(𝐱)
as in Equation <ref>, we specify a generative process for which p(𝐱) is a marginal distribution. Specifically, we first generate a continuous latent vector representation 𝐳 from a multivariate Gaussian prior p_θ(𝐳), and then generate the text sequence 𝐱



from a conditional distribution p_θ(𝐱 | 𝐳) parameterized using a neural net (often called the generation model or decoder). Because this model incorporates a latent variable that modulates the entire generation of each whole utterance, it may be better able to capture high-level sources of variation in the data. Specifically, in contrast with Equation <ref>, this generating distribution conditions
on latent vector representation 𝐳:

    p_θ(𝐱 |𝐳) = ∏_tp_θ(x_t | x_1, x_2, ..., x_t-1, 𝐳).


To estimate model parameters θ we
would ideally like to maximize the marginal probability
p_θ(𝐱) = ∫ p_θ(𝐳) p_θ(𝐱| 𝐳) d𝐳.
However, computing this marginal is intractable for many decoder choices. Thus, the following variational lower bound is often used as an objective <cit.>:

    log p_θ(𝐱) = -log∫  p_θ(𝐳) p_θ(𝐱| 𝐳) d𝐳
    
      
      
    
    
      ≥     _q_ϕ(𝐳|𝐱) [log p_θ(𝐱|𝐳)]
           - KL(q_ϕ(𝐳|𝐱) || p_θ(𝐳)).

Here, q_ϕ(𝐳 | 𝐱) is an approximation to the true posterior (often called the recognition model or encoder) and is parameterized by ϕ. Like the decoder, we have a choice of neural architecture to parameterize the encoder. However, unlike the decoder, the choice of encoder does not change the model class – it only changes the variational approximation used in training, which is a function of both the model parameters θ and the approximation parameters ϕ. Training seeks to optimize these parameters jointly using stochastic gradient ascent. A final wrinkle of the training procedure involves a stochastic approximation to the gradients of the variational objective (which is itself intractable). We omit details here, noting only that the final distribution of the posterior approximation q_ϕ(𝐳|𝐱) is typically assumed to be Gaussian so that a
re-parametrization trick can be used, and refer readers to  <cit.>.



 §.§ Training Collapse with Textual VAEs


Together, this combination of generative model and variational inference procedure are often referred to as a variational autoencoder (VAE).








We can also view the VAE as a regularized version of the autoencoder. Note, however, that while VAEs are valid probabilistic models whose likelihood can be evaluated on held-out data, autoencoders are not valid models. If only the first
term of the VAE variational bound _q_ϕ(𝐳|𝐱)[log
p_θ(𝐱|𝐳)] is used as an objective,
the variance of the posterior probability q_ϕ(𝐳|𝐱)
will become small and the training procedure reduces to an autoencoder.
It is the KL-divergence term,
KL(q_ϕ(𝐳|𝐱) || p_θ(𝐳)), that discourages the VAE memorizing each 𝐱 as a single
latent point.

While the KL term is critical for training VAEs, historically, instability on text has been evidenced by the KL term becoming vanishingly small during training, as observed by  <cit.>. When the training procedure collapses in this way, the result is an encoder that has duplicated the Gaussian prior (instead of a more interesting posterior), a decoder that completely ignores the latent variable 𝐳, and a learned model that reduces to a simpler language model. We hypothesize that this collapse condition is related to the contextual capacity of the decoder architecture. The choice encoder and decoder depends on the type of data. For images, these are typically MLPs or CNNs. LSTMs have been used for text, but have resulted in training collapse as discussed above <cit.>. Here, we propose to use a dilated CNN as the decoder instead. In one extreme, when the effective contextual width of a CNN is very large, it resembles the behavior of LSTM. When the width is very small, it behaves like a bag-of-words model. The architectural flexibility of dilated CNNs allows us to change the contextual capacity and conduct experiments to validate our hypothesis:
decoder contextual capacity and effective use of encoding information are directly related.
 We next describe the details of our decoder.


















 §.§ Dilated Convolutional Decoders 






The typical approach to using CNNs used for text generation <cit.> is
similar to that used for images <cit.>,
but with the convolution applied in one dimension. We take this approach here in defining our decoder.


One dimensional convolution: For a CNN to serve as a decoder for text, generation of x_t must only condition on past tokens x_<t. Applying the
traditional convolution will break this assumption and use tokens x_≥ t as inputs to predict
x_t. In our decoder, we avoid this by simply shifting the input by several slots <cit.>. With a convolution with filter size of k and using n layers,
our effective filter size (the number of past tokens to condition to in
predicting x_t) would be (k-1)× n + 1. Hence, the filter size would grow linearly with the depth of the network.


Dilation: Dilated convolution <cit.> was introduced to greatly increase the
effective receptive field size without increasing the computational cost. With
dilation d, the convolution is applied so that d-1 inputs are skipped each step.
Causal convolution can be seen a special case with d=1. With dilation, the effective
receptive size grows exponentially with network depth. In
Figure <ref>, we show dilation of sizes of 1 and 2 in the first and second layer, respectively. Suppose the dilation size in the i-th layer is d_i and we use the same filter
size k in all layers, then the effective filter size is (k-1)∑_id_i + 1.
The dilations are typically set to double every layer d_i+1 = 2d_i,
so the effective receptive field size can grow exponentially. Hence, the contextual capacity
of a CNN can be controlled across a greater range by manipulating the filter size, dilation size and network depth. We use this approach in experiments.


Residual connection:
We use residual connection <cit.> in the decoder
r0.13
  

  
    < g r a p h i c s >

  
  


to speed up
convergence and enable training of deeper models.
We use a residual block
(shown to the right)
similar to that
of <cit.>.
We use three convolutional layers with filter size 1×1, 1× k, 1× 1,
respectively, and ReLU activation
between convolutional layers.





Overall architecture: Our VAE architecture is shown in Figure <ref>.
We use LSTM as the encoder to get the posterior probability
q(𝐳|𝐱), which we assume to be
diagonal Gaussian. We parametrize the mean μ and
variance σ with LSTM output. We sample 𝐳 from
q(𝐳|𝐱), the decoder is conditioned on the sample by
concatenating 𝐳 with every word embedding of the decoder input.



 §.§ Semi-supervised VAE

In addition to conducting language modeling experiments, we will also conduct experiments on semi-supervised classification of text using our proposed decoder.
In this section, we briefly review


semi-supervised VAEs of <cit.> that incorporate discrete labels as additional variables.
Given the labeled set
(x, y) ∼ D_L and the unlabeled set
x ∼ D_U, <cit.> proposed a model
whose latent representation contains continuous vector 𝐳 and discrete label
𝐲:

    p(𝐱, 𝐲, 𝐳) =
      p(𝐲) p(𝐳) p(𝐱|𝐲, 𝐳).

The semi-supervised VAE fits a discriminative network
q(𝐲 | 𝐱), an inference network
q(𝐳| 𝐱, 𝐲) and a generative network
p(𝐱|𝐲,𝐳) jointly as part of optimizing a variational lower
bound similar that of basic VAE. For labeled data (𝐱, 𝐲),
this bound is:

    log p(𝐱, 𝐲) ≥   _q(𝐳|𝐱, 𝐲) [ log p(𝐱
                            | 𝐲, 𝐳)] 
       - KL(q(𝐳|𝐱, 𝐲) ||
                            p(𝐳)) + log p(𝐲) 
    
      =      L(𝐱, 𝐲) + log p(𝐲).

For unlabeled data 𝐱, the label is treated as a latent
variable, yielding:

    log p(𝐱) ≥    U(𝐱) 
    
      =    _q(𝐲|𝐱)[_q(𝐳|𝐱, 𝐲) [ log p(𝐱
          | 𝐲, 𝐳)]  
       - KL(q(𝐳|𝐱, 𝐲) ||
                         p(𝐳)) + log p(𝐲)   - log
                         q(𝐲|𝐱)] 
    
      =    ∑_yq(𝐲|𝐱) L(𝐱, 𝐲) -
          KL(q(𝐲|𝐱) || p(𝐲)).


Combining the labeled and unlabeled data terms, we have the overall objective
as:

    J =   _(𝐱, 𝐲) ∼ D_L[L(𝐱, 𝐲)] + _𝐱∼ D_U[U(𝐱)] 
        + α_(𝐱, 𝐲)∼ D_L[log q(𝐲|𝐱)],

where α controls the trade off between generative and
discriminative terms.





Gumbel-softmax: <cit.> propose
a continuous approximation to sampling from a categorical
distribution. Let u be a categorical distribution with
probabilities π_1, π_2, ..., π_c. Samples from u can be approximated using:

    y_i = exp((log (π_i) + g_i) / τ)/∑_j=1^cexp((log(π_j) + g_j)/τ),

where g_i follows Gumbel(0, 1).




The approximation is accurate when τ→ 0 and smooth when τ > 0.
In experiments, we use Gumbel-Softmax to approximate the samples from
p(𝐲|𝐱) to reduce the computational cost. As a result, we can directly
back propagate the gradients of U(𝐱) to the discriminator network.
We anneal τ so that sample variance is small
when training starts and then gradually decrease τ.

Unsupervised clustering:
In this section we adapt the same framework for
unsupervised clustering. We directly minimize the objective
U(𝐱), which is consisted of two parts: reconstruction loss and KL
regularization on q(𝐲|𝐱). The first part encourages the
model to assign 𝐱 to label 𝐲 such that the
reconstruction loss is low. We find that the model can easily get stuck in two
local optimum: the KL term is very small and q(𝐲 | 𝐱)
is close to uniform distribution or the KL term is very large and all
samples collapse to one class. In order to make the model more robust,
we modify the KL term by:

    KL_𝐲 = max(γ, KL(q(𝐲|𝐱) | p(𝐲)).

That is, we only minimize the KL term when it is large enough.



§ EXPERIMENTS



 §.§ Data sets


Since we would like to investigate VAEs for language modeling and
semi-supervised classification, the data sets should be suitable for
both purposes.






We use two large scale document classification data sets:
Yahoo Answer and Yelp15 review, representing topic classification
and sentiment classification data sets respectively
<cit.>.
The original data sets contain millions of samples, of which we sample
100k as training and 10k as validation and test from
the respective partitions. The detailed statistics of both data sets are in
Table <ref>. Yahoo Answer contains 10 topics including Society &
Culture, Science & Mathematics etc. Yelp15 contains 5 level of rating, with
higher rating better.







 §.§ Model configurations and Training details

We use an LSTM as an encoder for VAE and explore LSTMs and CNNs as decoders. For
CNNs, we explore several different configurations. We set the convolution filter
size to be 3 and gradually increase the depth and dilation from [1, 2, 4], [1,
2, 4, 8, 16] to [1, 2, 4, 8, 16, 1, 2, 4, 8, 16]. They represent small, medium
and large model and we name them as SCNN, MCNN and LCNN. We also explore a very large
model with dilations [1, 2, 4, 8, 16, 1, 2, 4, 8, 16, 1, 2, 4, 8, 16] and name it
as VLCNN. The effective filter size are 15, 63, 125 and 187 respectively. We
use the last hidden state of the encoder LSTM and feed it though an MLP to get the
mean and variance of q(𝐳|𝐱), from which we sample
𝐳 and then feed it through an MLP to get the starting state of decoder.
For the LSTM decoder, we follow <cit.> to use it as the initial state
of LSTM and feed it to every step of LSTM. For the CNN decoder, we concatenate it with the
word embedding of every decoder input.

The architecture of the Semi-supervised VAE basically follows that of the VAE. We
feed the last hidden state of the encoder LSTM through a two layer MLP then a
softmax to get q(𝐲|𝐱). We use Gumbel-softmax to
sample 𝐲 from q(𝐲|𝐱). We then concatenate
𝐲 with the last hidden state of encoder LSTM and feed them throught an
MLP to get the mean and variance of q(𝐳|𝐲,𝐱).
𝐲 and 𝐳 together are used as the starting state of the decoder.

We use a vocabulary size of 20k for both data sets and set the word embedding
dimension to be 512. The LSTM dimension is 1024. The number of channels for
convolutions in CNN decoders is 512 internally and 1024 externally,
as shown in Section <ref>.
We select the dimension of 𝐳 from [32,
64]. We find our model is not sensitive to this parameter.

We use Adam <cit.> to optimize all models and the learning rate is selected
from [2e-3, 1e-3, 7.5e-4] and β_1 is selected from [0.5, 0.9]. Empirically, we find
learning rate 1e-3 and β_1=0.5 to perform the best. We select drop out ratio of
LSTMs (both encoder and decoder) from [0.3, 0.5]. Following <cit.>,
we also use drop word for the LSTM decoder, the drop word ratio is selected from
[0, 0.3, 0.5, 0.7]. For the CNN decoder, we use a drop out ratio of 0.1 at
each layer. We do not use drop word for CNN decoders. We use batch size of 32
and all model are trained for 40 epochs. We start to half the learning rate
every 2 epochs after epoch 30. Following <cit.>, we use KL cost
annealing strategy. We set the initial weight of KL cost term to be 0.01 and
increase it linearly until a given iteration T. We treat T as a hyper parameter
and select it from [10k, 40k, 80k].



 §.§ Language modeling results


The results for language modeling are shown in Table <ref>. We report
the negative log likelihood (NLL) and perplexity (PPL) of the test set. For the NLL of VAEs,
we decompose it into reconstruction loss and KL divergence and report the
KL divergence in the parenthesis. To better visualize these results, we plot
the results of Yahoo data set (Table <ref>) in Figure <ref>.

We first look at the LM results for Yahoo data set. As we gradually increase
the effective filter size of CNN from SCNN, MCNN to LCNN, the NLL decreases from
345.3, 338.3 to 335.4. The NLL of LCNN-LM is very close to the NLL of LSTM-LM
334.9. But VLCNN-LM is a little bit worse than LCNN-LM, this indicates a little
bit of over-fitting.

We can see that LSTM-VAE is worse than LSTM-LM in terms of NLL and the KL
term is nearly zero, which verifies the finding
of <cit.>. When we use CNNs as the decoders for VAEs,
we can see improvement over pure CNN LMs. For SCNN, MCNN and LCNN,
the VAE results improve over LM results from 345.3 to 337.8, 338.3 to 336.2, and 335.4 to 333.9
respectively. The improvement is big for small models and gradually decreases
as we increase the decoder model contextual capacity. When the model is as large as VLCNN,
the improvement diminishes and the VAE result is almost the same with LM result.
This is also reflected in the KL term, SCNN-VAE has
the largest KL of 13.3 and VLCNN-VAE has the smallest KL of 0.7.
When LCNN is used as the decoder, we obtain an optimal trade off between using
contextual information and latent representation. LCNN-VAE achieves a
NLL of 333.9, which improves over LSTM-LM with NLL of 334.9.

We find that if we initialize the parameters of LSTM encoder with
parameters of LSTM language model, we can improve the VAE results further.
This indicates better encoder model is also a key factor for VAEs
to work well. Combined with encoder initialization, LCNN-VAE improves over LSTM-LM from
334.9 to 332.1 in NLL and from 66.2 to 63.9 in PPL. Similar results for the sentiment data set are shown in Table <ref>. LCNN-VAE improves over LSTM-LM from 362.7
to 359.1 in NLL and from 42.6 to 41.1 in PPL.




Latent representation visualization: In order to visualize the latent representation,
we set the dimension of 𝐳 to be 2
and plot the mean of posterior probability q(𝐳|𝐱), as shown in
Figure <ref>. We can see distinct different characteristics of topic
and sentiment representation. In Figure <ref>, we can see that documents
of different topics fall into different clusters, while in
Figure <ref>, documents of different ratings
form a continuum, they lie continuously on the x-axis as the review rating increases.








 §.§ Semi-supervised VAE results

Motivated by the success of VAEs for language modeling, we continue to explore
VAEs for semi-supervised learning. Following that of <cit.>,
we set the number of labeled samples to be 100, 500, 1000 and 2000 respectively.


Ablation Study: At first, we would like to explore the effect of
different decoders for semi-supervised classification.
We fix the number of labeled samples to be 500 and report
both classification accuracy and NLL of the test set of Yahoo data set in
Table. <ref>. We can see that SCNN-VAE-Semi has the best
classification accuracy of 65.5. The accuracy decreases as we gradually increase the
decoder contextual capacity. On the other hand, LCNN-VAE-Semi
has the best NLL result. This classification accuracy and NLL trade off
once again verifies our conjecture: with small contextual window size,
the decoder is forced to use the encoder information, hence the latent
representation is better learned.

Comparing the NLL results of Table <ref>
with that of Table <ref>, we can see the
NLL improves. The NLL of semi-supervised VAE improves over simple
VAE from 337.8 to 335.7 for SCNN, from 336.2 to 332.8 for MCNN,
and from 333.9 to 332.8 for LCNN.
The improvement mainly comes from the KL divergence part, this indicates that better latent
representations decrease the KL divergence, further improving
the VAE results.


Comparison with related methods: We compare Semi-supervised VAE with the
methods from <cit.>, which represent the previous state-of-the-art for semi-supervised
sequence learning. <cit.> pre-trains a classifier by initializing the parameters
of a classifier with that of a language model or a sequence autoencoder. They find it improves
the classification accuracy significantly. Since SCNN-VAE-Semi performs the best
according to Table <ref>, we fix
the decoder to be SCNN in this part. The detailed comparison is in
Table <ref>. We can see that semi-supervised VAE performs better than LM-LSTM and
LA-LSTM from <cit.>. We also initialize the encoder of the VAE with parameters
from LM and find classification accuracy further improves.
We also see the advantage of SCNN-VAE-Semi
over LM-LSTM is greater when the number of labeled samples is smaller. The
advantage decreases as we increase the number of labeled samples. When we
set the number of labeled samples to be 25k, the SCNN-VAE-Semi achieves an accuracy of 70.4,
which is similar to LM-LSTM with an accuracy of 70.5. Also, SCNN-VAE-Semi performs better on Yahoo
data set than Yelp data set. For Yelp, SCNN-VAE-Semi is a little bit worse than LM-LSTM
if the number of labeled samples is greater than 100, but becomes better when we initialize the encoder.
Figure <ref> explains this observation. It shows the documents are coupled
together and are harder to classify. Also, the latent representation contains
information other than sentiment, which may not be useful for classification.


 §.§ Unsupervised clustering results























We also explored using the same framework for unsupervised
clustering. We compare with the baselines that extract the feature with
existing models and then run Gaussian Mixture Model (GMM) on these features. We
find empirically that simply using the features does not perform well since the
features are high dimensional. We run a PCA on these features, the dimension of
PCA is selected from [8, 16, 32]. Since GMM can easily get stuck in poor local
optimum, we run each model ten times and report the best result.
We find directly optimizing U(𝐱) does not perform well for unsupervised
clustering and we need to initialize the encoder with LSTM language model. The
model only works well for Yahoo data set. This is potentially because
Figure <ref> shows that sentiment latent representations does
not fall into clusters. γ in Equation <ref> is a sensitive parameter,
we select it from the range between 0.5 and 1.5 with an interval of 0.1.
We use the following evaluation protocol <cit.>: after we
finish training, for cluster i, we find out the validation sample 𝐱_n from cluster i
that has the best q(y_i|𝐱) and assign the label of 𝐱_n to all samples in
cluster i. We then compute the test accuracy based on this assignment.
The detailed results are in Table <ref>. We can see
SCNN-VAE-Unsup + init performs better than other baselines. LSTM+GMM performs very
bad probably because the feature dimension is 1024 and is too high for
GMM, even though we already used PCA to reduce the dimension.


Conditional text generation
With the semi-supervised VAE, we are able to generate text conditional on the
label. Due to space limitation, we only show one example of generated reviews
conditioning on review rating in Table <ref>.


For
each group of generated text, we fix 𝐳 and vary the label 𝐲, while picking 𝐱 via beam search with a beam size of 10.



§ RELATED WORK

Variational inference via the re-parameterization trick was initially proposed
by <cit.> and since then, VAE has been
widely adopted as generative model for images <cit.>.

Our work is in line with previous works on combining variational inferences
with text
modeling <cit.>. <cit.> is the first work to
combine VAE with language model and they use LSTM as the decoder and find some
negative results. On the other hand, <cit.> models text as bag
of words, though improvement has been found, the model can not be used to
generate text.  Our work fills the gaps between
them. <cit.> applies variational
inference to dialogue modeling and machine translation and found some
improvement in terms of generated text quality, but no language modeling
results are
reported. <cit.>
embedded variational units in every step of a RNN, which is different from our model
in using global latent variables to learn high level features.

Our use of CNN as decoder is inspired by recent success of PixelCNN model for
images <cit.>, WaveNet for audios <cit.>,
Video Pixel Network for video modeling <cit.> and ByteNet
for machine translation <cit.>. But in contrast to those
works showing using a very deep architecture leads to better performance, CNN
as decoder is used in our model to control the contextual capacity, leading to better performance.

Our work is closed related the recently proposed variational lossy
autoencoder <cit.> which is used to predict image
pixels. They find that conditioning on a smaller window of a pixels leads to better
results with VAE, which is similar to our finding.
Much <cit.> has been done to
come up more powerful prior/posterior distribution representations with techniques such as
normalizing flows. We treat this as one of our future works.
This work is largely orthogonal and could be potentially
combined with a more effective choice of decoder to yield additional gains.

There is much previous work exploring unsupervised sentence encodings, for example
skip-thought vectors <cit.>, paragraph
vectors <cit.>, and sequence
autoencoders <cit.>. <cit.> applies a pretrained
model to semi-supervised classification and find significant gains, we use this
as the baseline for our semi-supervised VAE.




§ CONCLUSION


We showed that by controlling the decoder's contextual capacity in VAE, we can improve performance on both language modeling and semi-supervised classification tasks by preventing a degenerate collapse of the training procedure. These results indicate that more carefully characterizing decoder capacity and understanding how it relates to common variational training procedures may represent important avenues for unlocking future unsupervised problems.
















icml2017







