Question: Represent the scientific article to find a short TLDR

Question: Amortized inference has led to efficient approximate inference for large datasets. The quality of posterior inference is largely determined by two factors: a) the ability of the variational distribution to model the true posterior and b) the capacity of the recognition network to generalize inference over all datapoints. We analyze approximate inference in variational autoencoders in terms of these factors. We find that suboptimal inference is often due to amortizing inference rather than the limited complexity of the approximating distribution. We show that this is due partly to the generator learning to accommodate the choice of approximation. Furthermore, we show that the parameters used to increase the expressiveness of the approximation play a role in generalizing inference rather than simply improving the complexity of the approximation. There has been significant work on improving inference in variational autoencoders (VAEs) BID13 BID22 through the development of expressive approximate posteriors BID21 BID14 BID20 BID27. These works have shown that with more expressive approximate posteriors, the model learns a better distribution over the data. In this paper, we analyze inference suboptimality in VAEs: the mismatch between the true and approximate posterior. In other words, we are interested in understanding what factors cause the gap between the marginal log-likelihood and the evidence lower bound (ELBO). We refer to this as the inference gap. Moreover, we break down the inference gap into two components: the approximation gap and the amortization gap. The approximation gap comes from the inability of the approximate distribution family to exactly match the true posterior. The amortization gap refers to the difference caused by amortizing the variational parameters over the entire training set, instead of optimizing for each datapoint independently. We refer the reader to Table 1 for detailed definitions and FIG0 for a simple illustration of the gaps. In FIG0, L[q] refers to the ELBO using an amortized distribution q, whereas q * is the optimal q within its variational family. Our experiments investigate how the choice of encoder, posterior approximation, decoder, and model optimization affect the approximation and amortization gaps. We train VAE models in a number of settings on the MNIST, Fashion-MNIST BID30, and CIFAR-10 datasets. Our contributions are: a) we investigate inference suboptimality in terms of the approximation and amortization gaps, providing insight to guide future improvements in VAE inference, b) we quantitatively demonstrate that the learned true posterior accommodates the choice of approximation, and c) we demonstrate that using parameterized functions to improve the expressiveness of the approximation plays a large role in reducing error caused by amortization. Table 1: Summary of Gap Terms. The middle column refers to the general case where our variational objective is a lower bound on the marginal log-likelihood (not necessarily the ELBO). The right most column demonstrates the specific case in VAEs. q * (z|x) refers to the optimal approximation within a family Q, i.e. q * (z|x) = arg min q∈Q KL (q(z|x)||p(z|x)). Let x be the observed variable, z the latent variable, and p(x, z) be their joint distribution. Given a dataset X = {x 1, x 2, ..., x N}, we would like to maximize the marginal log-likelihood: DISPLAYFORM0 In practice, the marginal log-likelihood is computationally intractable due to the integration over the latent variable z. Instead, VAEs optimize the ELBO of the marginal log-likelihood BID13 BID22: DISPLAYFORM1 ≥ E z∼q(z|x) log p(x, z) q(z|x) = L VAE [q].From the above we can see that the lower bound is tight if q(z|x) = p(z|x). The choice of q(z|x) is often a factorized Gaussian distribution for its simplicity and efficiency. VAEs perform amortized inference by utilizing a recognition network (encoder), ing in efficient approximate inference for large datasets. The overall model is trained by stochastically optimizing the ELBO using the reparametrization trick BID13. There are a number of strategies for increasing the expressiveness of approximate posteriors, going beyond the original factorized-Gaussian. We briefly summarize normalizing flows and auxiliary variables. Normalizing flow BID21 ) is a change of variables procedure for constructing complex distributions by transforming probability densities through a series of invertible mappings. Specifically, if we transform a random variable z 0 with distribution q 0 (z), the ing random variable z T = T (z 0) has a distribution: DISPLAYFORM0 By successively applying these transformations, we can build arbitrarily complex distributions. Stacking these transformations remains tractable due to the determinant being decomposable: det(AB) = det(A)det(B). An important property of these transformations is that we can take expectations with respect to the transformed density q T (z T) without explicitly knowing its formula known as the law of the unconscious statistician (LOTUS): DISPLAYFORM1 Using the change of variable and LOTUS, the lower bound can be written as: DISPLAYFORM2 The main constraint on these transformations is that the determinant of their Jacobian needs to be easily computable. Deep generative models can be extended with auxiliary variables which leave the generative model unchanged but make the variational distribution more expressive. Just as hierarchical Bayesian models induce dependencies between data, hierarchical variational models can induce dependencies between latent variables. The addition of the auxiliary variable changes the lower bound to: DISPLAYFORM0 where r(v|x, z) is called the reverse model. From Eqn. 8, we see that this bound is looser than the regular ELBO, however the extra flexibility provided by the auxiliary variable can in a higher lower bound. This idea has been employed in works such as auxiliary deep generative models (ADGM,), hierarchical variational models (HVM, BID20) and Hamiltonian variational inference (HVI, BID25). We use two bounds to estimate the marginal log-likelihood of a model: IWAE BID2 and AIS BID19 ). Here we describe the IWAE bound. See Section 6.5 in the appendix for a description of AIS.The IWAE bound is a tighter lower bound than the VAE bound. More specifically, if we take multiple samples from the q distribution, we can compute a tighter lower bound on the marginal log-likelihood: DISPLAYFORM0 As the number of importance samples approaches infinity, the bound approaches the marginal loglikelihood. This importance weighted bound was introduced along with the Importance Weighted Autoencoder BID2, thus we refer to it as the IWAE bound. It is often used as an evaluation metric for generative models BID2 BID14. As shown by BID0 and BID4, the IWAE bound can be seen as using the VAE bound but with an importance weighted q distribution. The inference gap G is the difference between the marginal log-likelihood log p(x) and a lower bound L [q]. Given the distribution in the family that maximizes the bound, q * (z|x) = arg max q∈Q L[q], the inference gap decomposes as the sum of approximation and amortization gaps: DISPLAYFORM0 For VAEs, we can translate the gaps to KL divergences by rearranging: DISPLAYFORM1 3.2 FLEXIBLE APPROXIMATE POSTERIOR Our experimentation compares two families of approximate posteriors: the fully-factorized Gaussian (FFG) and a flexible flow (Flow). Our choice of flow is a combination of the Real NVP BID5 and auxiliary variables BID20. Our model also resembles leap-frog dynamics applied in Hamiltonian Monte Carlo (HMC, BID18).Let z ∈ R n be the variable of interest and v ∈ R n the auxiliary variable. Each flow step involves: DISPLAYFORM2 where σ 1, σ 2, µ 1, µ 2: R n → R n are differentiable mappings parameterized by neural nets and • takes the Hadamard or element-wise product. The determinant of the combined transformation's Jacobian, |det(Df)|, can be easily evaluated. See section 6.2 in the Appendix for a detailed derivation. Thus, we can jointly train the generative and flow-based inference model by optimizing the bound: DISPLAYFORM3 Additionally, multiple such type of transformations can be stacked to improve expressiveness. We refer readers to section 6.1.2 in the Appendix for details of our flow configuration adopted in the experimentation. We use several bounds to compute the inference gaps. To estimate the marginal log-likelihood, logp(x), we take the maximum of our tightest lower bounds, specifically the maximum between the IWAE and AIS bounds. To compute the AIS bound, we use 100 chains, each with 500 intermediate distributions, where each transition consists of one HMC trajectory with 10 leapfrog steps. The initial distribution for AIS is the prior, so that it is encoder-independent. For our experiments, we test two different variational distributions: the fully-factorized Gaussian q F F G and the flexible approximation q F low as described in section 3.2. When computing DISPLAYFORM0, we use 5000 samples. To compute L VAE [q *], we optimize the parameters of the variational distribution for every datapoint. See Section 6.4 for details of the local optimization and stopping criteria. Much of the earlier work on variational inference focused on optimizing the variational parameters locally for each datapoint, e.g. the original Stochastic Variational Inference scheme specifies the variational parameters to be optimized locally in the inner loop. BID24 perform such local optimization when learning deep Boltzmann machines. More recent work has applied this idea to improve approximate inference in directed Belief networks BID9.Most relevant to our work is the recent work of BID15. They explicitly remark on two sources of error in variational learning with inference networks, and propose to optimize approximate inference locally from an initialization output by the inference network. They show improved training on high-dimensional, sparse data with the hybrid method, claiming that local optimization reduces the negative effects of random initialization in the inference network early on in training. Yet, their work only dwells on reducing the amortization gap and does analyze the error arising from the use of limited approximating distributions. Even though it is clear that failed inference would lead to a failed generative model, little quantitative assessment has been done showing the effect of the approximate posterior on the true posterior. BID2 visually demonstrate that when trained with an importance-weighted approximate posterior, the ing true posterior is more complex than those trained with fully-factorized Gaussian approximations. We extend this observation quantitatively in the setting of flow-based approximate inference. To begin, we would like to gain some insight into the properties of inference in VAEs by visualizing different distributions in the latent space. To this end, we trained a VAE with a two-dimensional latent space on MNIST. We show contour plots of various distributions in the latent space in FIG1. The first row contains contour plots of the true posteriors p(z|x) for four different training datapoints (columns). We have selected these four examples to highlight different inference phenomena. The amortized FFG row refers to the output of the recognition net, in this case, a fully-factorized Gaussian (FFG) approximation. Optimal FFG is the FFG that best fits the posterior of the datapoint. Optimal Flow is the optimal fit of a flexible distribution to the same posterior, where the flexible distribution we use is described in Section 3.2.Posterior A is an example of a distribution where FFG can fit well. Posterior B is an example of dependence between dimensions, demonstrating the limitation of having a factorized approximation. Posterior C highlights a shortcoming of performing amortization with a limited-capacity recognition network, where the amortized FFG shares little support with the true posterior. Posterior D is a bimodal distribution which demonstrates the ability of the flexible approximation to fit to complex distributions, in contrast to the simple FFG approximation. These observations raise the following question: in more typical VAEs, is the amortization of inference the leading cause of the distribution mismatch, or is it the choice of approximation? Table 2: Inference Gaps. The columns q F F G and q F low refer to the variational distribution used for training the model. All numbers are in nats. DISPLAYFORM0 Here we will compare the influence that the approximation and amortization errors have on the total inference gap. Table 2 are from training on MNIST, Fashion-MNIST and CIFAR-10.For each dataset, we trained two different approximate posterior distributions: a fully-factorized Gaussian, q F F G, and a flexible distribution, q F low. Due to the computational cost of optimizing the local parameters for each datapoint, our evaluation is performed on a subset of 1000 datapoints for MNIST and Fashion-MNIST and a subset of 100 datapoints for CIFAR-10.For MNIST, we see that the amortization and approximation gaps each account for nearly half of the inference gap. On Fashion-MNIST, which is a more difficult dataset to model, the amortization gap becomes larger than the approximation gap. Similarly for CIFAR-10, we see that the amortization gap is much more significant than the approximation gap. Thus, for the three datasets and model architectures that we tested, the amortization gap seems to be the prominent cause of inference suboptimality, especially when the difficulty of the dataset increases. This analysis indicates that improvements in inference will likely be a of reducing amortization error, rather than approximation errors. With these in mind, would simply increasing the capacity of the encoder improve the amortization gap? We examined this by training the MNIST and Fashion-MNIST models from above but with larger encoders. See Section 6.1.2 for implementation details. Table 3 are the of this experiment. Comparing to Table 2, we see that for both datasets and both variational distributions, the inference gap decreases and the decrease is mainly due to a reduction in the amortization gap. Table 3: Larger Encoder. The columns q F F G and q F low refer to the variational distribution used for training the model. All numbers are in nats. DISPLAYFORM0 The common reasoning for increasing the expressiveness of the approximate posterior is to minimize the difference between the true and approximate, i.e. reduce the approximation gap. However, given that the expressive approximation is often accompanied by many additional parameters, we would like to know if it has an influence on the amortization error. To investigate this, we trained a VAE in the same manner as Section 5.2. After training, we kept the generator fixed and trained new encoders to fit to the fixed posterior. Specifically, we trained a small encoder with a factorized Gaussian q distribution to obtain a large amortization gap. We then trained a small encoder with a flow distribution. See Section 6.2 for the details of the experiment. The are shown in TAB3. As expected, we observe that the small encoder has a very large amortization gap. However, when we use q F low as the approximate distribution, we see the approximation gap decrease, but more importantly, there is a significant decrease in the amortization gap. This indicates that the parameters used for increasing the complexity of the approximation also play a large role in diminishing the amortization error. These are expected given that the parameterization of the Flow distribution can be interpreted as an instance of the RevNet BID7 which has demonstrated that Real-NVP like transformations BID5 can model complex functions similar to typical MLPs. Thus the flow transformations we employ should also be expected to increase the expressiveness while also increasing the capacity of the encoder. The implication of this observation is that models which improve the flexibility of their variational approximation, and attribute their improved to the increased expressiveness, may have actually been due to the reduction in amortization error. We have seen that increasing the expressiveness of the approximation improves the marginal likelihood of the trained model, but to what amount does it alter the true posterior? Will a factorized Gaussian approximation cause the true posterior to be more like a factorized Gaussian or is the true posterior mostly fixed? Just as it is hard to evaluate a generative model by visually inspecting samples from the model, its hard to say how Gaussian the true posterior is by visual inspection. We can quantitatively determine how close the posterior is to a fully factorized Gaussian (FFG) distribution by comparing the marginal log-likelihood estimate, logp(x), and the Optimal FFG bound, DISPLAYFORM0 In other words, we are estimating the KL divergence between the optimal Gaussian and the true posterior, KL (q * (z|x)||p(z|x)).In Table 2 on MNIST, the Optimal Flow improves upon the Optimal FFG for the FFG trained model by 0.4 nats. In contrast, on the Flow trained model, the difference increases to 12.5 nats. This suggests that the true posterior of a FFG-trained model is closer to FFG than the true posterior of the Flow-trained model. The same observation can be made on the Fashion-MNIST dataset. This implies that the decoder can learn to have a true posterior that fits better to the approximation. Although the generative model can learn to have a posterior that fits to the approximation, it seems that not having this constraint, ie. using a flexible approximate, in better generative models. We can use these observations to help justify our approximation and amortization gap of Section 5.2. Those showed that the amortization error is often the main cause of inference suboptimality. One reason for this is that the generator accommodates to the choice of approximation, as shown above, thus reducing the approximation error. Given that we have seen that the generator could accommodate to the choice of approximation, our next question is whether a generator with more capacity can accommodate more. To this end, we trained VAEs with decoders of different sizes and measured the approximation gaps. Specifically, we trained decoders with 0, 2, and 4 hidden layers on MNIST. See Table 5 for the . We see that as the capacity of the decoder increases, the approximation gap decreases. This implies that the more flexible the generator, the less flexible the approximate distribution needs to be. Table 5: Increased decoder capacity reduces approximation gap. All numbers are in nats. Typical warm-up BID1 refers to annealing KL (q(z|x)||p(z)) during training. This can also be interpreted as performing maximum likelihood estimation (MLE) early on during training. This optimization technique is known to help prevent the latent variable from degrading to the prior BID2. We employ a similar annealing scheme during training. Rather than annealing the KL divergence, we anneal the entropy of the approximate distribution q: DISPLAYFORM0 where λ is annealed from 0 to 1 over training. This can be interpreted as maximum a posteriori (MAP) in the initial phase. Due to its similarity, we will also refer to this technique as warm-up. We find that warm-up techniques, such as annealing the entropy, are important for allowing the true posterior to be more complex. Table 6 are from a model trained without the entropy annealing schedule. Comparing these to Table 2, we observe that the difference between DISPLAYFORM1 F low ] is significantly smaller without entropy annealing. This indicates that the true posterior is more Gaussian when entropy annealing is not used. This suggests that, in addition to preventing the latent variable from degrading to the prior, entropy annealing allows the true posterior to better utilize the flexibility of the expressive approximation, ing in a better trained model. Table 6: Models trained without entropy annealing. The columns q F F G and q F low refer to the variational distribution used for training the model. All numbers are in nats. DISPLAYFORM2 In this paper, we investigated how encoder capacity, approximation choice, decoder capacity, and model optimization influence inference suboptimality in terms of the approximation and amortization gaps. We found that the amortization gap is often the leading source of inference suboptimality and that the generator reduces the approximation gap by learning a true posterior that fits to the choice of approximate distribution. We showed that the parameters used to increase the expressiveness of the approximation play a role in generalizing inference rather than simply improving the complexity of the approximation. We confirmed that increasing the capacity of the encoder reduces the amortization error. We also showed that optimization techniques, such as entropy annealing, help the generative model to better utilize the flexibility of the expressive variational distribution. Computing these gaps can be useful for guiding improvements to inference in VAEs. Future work includes evaluating other types of expressive approximations and more complex likelihood functions. The VAE model of FIG1 uses a decoder p(x|z) with architecture: 2 − 100 − 784, and an encoder q(z|x) with architecture: 784 − 100 − 4. We use tanh activations and a batch size of 50. The model is trained for 3000 epochs with a learning rate of 10 −4 using the ADAM optimizer BID12. Both MNIST and Fashion-MNIST consist of a training and test set with 60k and 10k datapoints respectively, where each datapoint is a 28x28 grey-scale image. We rescale the original images so that pixel values are within the range. For MNIST, We use the statically binarized version described by BID16. We also binarize Fashion-MINST statically. For both datasets, we adopt the Bernoulli likelihood for the generator. The VAE models for MNIST and Fashion-MNIST experiments have the same architecture given in table 7. The flow configuration is given in table 8. Generator Input ∈ R Table 7: Neural net architecture for MNIST/Fashion-MNIST experiments. In the large encoder setting, we change the number of hidden units for the inference network to be 500, instead of 200. The warm-up models are trained with a linear schedule over the first 400 epochs according to Section 5.3.1.The activation function is chosen to be the exponential linear unit (ELU, BID3), as we observe improved performance compared to tanh. We follow the same learning rate schedule and train for the same amount of epochs as described by BID2. All models are trained with the a batch-size of 100 with ADAM. CIFAR-10 consists of a training and test dataset with 50k and 10k datapoints respectively, where each datapoint is a 32 × 32 color image. We rescale individual pixel values to be in the range. We follow the discretized logistic likelihood model adopted by BID14, where each input channel has its own scale learned by an MLP. For the latent variable, we use a 32-dimensional factorized Gaussian for q(z|x) following BID14. For all neural networks, ELU is chosen to be the activation function. The specific network architecture is shown in Table 9.We adopt a gradually decreasing learning rate with an initialize value of 10 −3. Warm-up is applied with a linear schedule over the first 20 epochs. All models are trained with a batch-size of 100 with ADAM. Early-stopping is applied based on the performance on the held-out set. For the model with expressive inference, we use four flow steps as opposed to only two in MNIST/Fashion-MNIST experiments. The aim of this experiment is to show that the parameters used for increasing the expressiveness of the approximation also contribute to reducing the amortization error. To show this, we train a VAE on MNIST, discard the encoder, then retrain two encoders on the fixed decoder: one with a factorized Gaussian distribution and the other with a parameterized'flow' distribution. We use fixed decoder so that the true posterior is constant for both encoders. Next, we describe the encoders which were trained on the fixed trained decoder. In order to highlight a large amortization gap, we employed a very small encoder architecture: D X − 2D Z. This encoder has no hidden layers, which greatly impoverishes its ability and in a large amortization gap. We compare two approximate distributions q(z|x). Firstly, we experiment with the typical fully factorized Gaussian (FFG). The second is what we call a flow distribution. Specifically, we use the transformations of BID5. We also include an auxiliary variable so we don't need to select how to divide the latent space for the transformations. The approximate distribution over the latent z and auxiliary variable v factorizes as: q(z, v|x) = q(z|x)q(v). The q(v) distribution is simply a N distribution. Since we're using a auxiliary variable, we also require the r(v|z) distribution which we parameterize as r(v|z): [D Z] − 50 − 50 − 2D Z. The flow transformation is the same as in Section 3.2, which we apply twice. DISPLAYFORM0 FC. 100-ELU-FC. 100-ELU-FC. 50+50 FC. 100-ELU-FC. 100-ELU-FC. 50+50 DISPLAYFORM1 FC. 100-ELU-FC. 100-ELU-FC. 50 FC. 100-ELU-FC. 100-ELU-FC. 50 Table 9: Network architecture for CIFAR-10 experiments. For the generator, one of the MLPs immediately after the input layer of the generator outputs channel-wise scales for the discretized logistic likelihood model. BN stands for batch-normalization. The overall mapping f that performs (z, v) → (z, v) is the composition of two sheer mappings f 1 and f 2 that respectively perform (z, v) → (z, v) and (z, v) → (z, v). Since the Jacobian of either one of the sheer mappings is diagonal, the determinant of the composed transformation's Jacobian Df can be easily computed: DISPLAYFORM0 For the local FFG optimization, we initialize the mean and variance as the prior, i.e. N (0, I). We optimize the mean and variance using the Adam optimizer with a learning rate of 10 −3. To determine convergence, after every 100 optimization steps, we compute the average of the previous 100 ELBO values and compare it to the best achieved average. If it does not improve for 10 consecutive iterations then the optimization is terminated. For the Flow model, the same process is used to optimize all of its parameters. All neural nets for the flow were initialized with a variant of the Xavier initilization BID6. We use 100 Monte Carlo samples to compute the ELBO to reduce variance. Annealed importance sampling (; BID11) is a means of computing a lower bound to the marginal log-likelihood. Similarly to the importance weighted bound, AIS must sample a proposal distribution f 1 (z) and compute the density of these samples, however, AIS then transforms the samples through a sequence of reversible transitions T t (z |z). The transitions anneal the proposal distribution to the desired distribution f T (z).Specifically, AIS samples an initial state z 1 ∼ f 1 (z) and sets an initial weight w 1 = 1. For the following annealing steps, z t is sampled from T t (z |z) and the weight is updated according to: DISPLAYFORM0.This procedure produces weight w T such that E [w T] = Z T /Z 1, where Z T and Z 1 are the normalizing constants of f T (z) and f 1 (z) respectively. This pertains to estimating the marginal likelihood when the target distribution is p(x, z) when we integrate with respect to z. Typically, the intermediate distributions are simply defined to be geometric averages: DISPLAYFORM1 βt, where β t is monotonically increasing with β 1 = 0 and β T = 1. When f 1 (z) = p(z) and f T (z) = p(x, z), the intermediate distributions are: DISPLAYFORM2 Model evaluation with AIS appears early on in the setting of deep belief networks BID23. AIS for decoder-based models was also used by BID29. They validated the accuracy of the approach with Bidirectional Monte Carlo (BDMC, BID8) and demonstrated the advantage of using AIS over the IWAE bound for evaluation when the inference network overfits to the training data. How well is inference done in VAEs during training? Are we close to doing the optimal or is there much room for improvement? To answer this question, we quantitatively measure the inference gap: the gap between the true marginal log-likelihood and the lower bound. This amounts to measuring how well inference is being done during training. Since we cannot compute the exact marginal log-likelihood, we estimate it using the maximum of any of its lower bounds, described in 3.3. Fig. 3a shows training curves for a FFG and Flow inference network as measured by the VAE, IWAE, and AIS bounds on the training and test set. The inference gap on the training set with the FFG model is 3.01 nats, whereas the Flow model is 2.71 nats. Accordingly, Fig. 3a shows that the training IWAE bound is slightly tighter for the Flow model compared to the FFG. Due to this lower inference gap during training, the Flow model achieves a higher AIS bound on the test set than the FFG model. To demonstrate that a very small inference gap can be achieved, even with a limited approximation such as a factorized Gaussian, we train the model on a small dataset. In this experiment, our training set consists of 1000 datapoints randomly chosen from the original MNIST training set. The training curves on this small datatset are show in Fig. 3b. Even with a factorized Gaussian distribution, the inference gap is very small: the AIS and IWAE bounds are overlapping and the VAE is just slightly below. Yet, the model is overfitting as seen by the decreasing test set bounds. Figure 3: Training curves for a FFG and a Flow inference model on MNIST. AIS provides the tightest lower bound and is independent of encoder overfitting. There is little difference between FFG and Flow models trained on the 1000 datapoints since inference is nearly equivalent. We will begin by explaining how we separate encoder from decoder overfitting. Decoder overfitting is the same as in the regular supervised learning scenario, where we compare the train and test error. To measure decoder overfitting independently from encoder overfitting, we use the AIS bound since it is encoder-independent. Thus we can observe decoder overfitting through the AIS test training curve. In contrast, the encoder can only overfit in the sense that the recognition network becomes unsuitable for computing the marginal likelihood on the test set. Thus, encoder overfitting is computed by: L AIS − L IW on the test set. For the small dataset of Fig. 3b, it clear that there is significant encoder and decoder overfitting. A model trained in this setting would benefit from regularization. For Fig. 3a, the model is not overfit and would benefit from more training. However, there is some encoder overfitting due to the gap between the AIS and IWAE bounds on the test set. Comparing the FFG and Flow models, it appears that the Flow does not have a large effect on encoder or decoder overfitting. The flexiblity of the Gaussian family with arbitrary covariance lies between that of FFG and Flow. With covariance, the Gaussian distribution can model interactions between different latent dimensions. Yet, compared to Flow, its expressiveness is limited due to its inability to model higher order interactions and its unimodal nature. To apply the reparameterization trick, we perform the Cholesky decomposition on the covariance matrix: Σ = LL, where L is lower triangular. A sample from N (µ, Σ) could be obtained by first sampling from a unit Gaussian ∼ N (0, I), then computing z = µ + L.To analyze the capability of the Gaussian family, we train several VAEs on MNIST and Fashion-MNIST with the approximate posterior q(z|x) being a Gaussian with full covariance. To inspect how well inference is done, we perform the local optimizations described in Section 5. Table 10: Gaussian latents trained with full covariance. We can see from table 10 that local optimization with FFG on a model trained with full covariance inference produces a bad lower bound. This resonates with the argument that the approximation has a significant influence on the true posterior as described in section 5.3.Comparing to numbers in table 2, we can see that the full-covariance VAE trained on MNIST is nearly on par with that trained with Flow (-89.28 vs -88.94). For Fashion-MNIST, the fullcovariance VAE even performs better by a large margin in terms of the estimated log-likelihood (-96.46 vs -97.41).

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We decompose the gap between the marginal log-likelihood and the evidence lower bound and study the effect of the approximate posterior on the true posterior distribution in VAEs.

This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.


Question: Represent the scientific article to find a short TLDR

Question: In this paper, we propose a framework that leverages semi-supervised models to improve unsupervised clustering performance. To leverage semi-supervised models, we first need to automatically generate labels, called pseudo-labels. We find that prior approaches for generating pseudo-labels hurt clustering performance because of their low accuracy. Instead, we use an ensemble of deep networks to construct a similarity graph, from which we extract high accuracy pseudo-labels. The approach of finding high quality pseudo-labels using ensembles and training the semi-supervised model is iterated, yielding continued improvement. We show that our approach outperforms state of the art clustering for multiple image and text datasets. For example, we achieve 54.6% accuracy for CIFAR-10 and 43.9% for 20news, outperforming state of the art by 8-12% in absolute terms. Semi-supervised methods, which make use of large unlabelled data sets and a small labelled data set, have seen recent success, e.g., ladder networks achieves 99% accuracy in MNIST using only 100 labelled samples. These approaches leverage the unlabelled data to help the network learn an underlying representation, while the labelled data guides the network towards separating the classes. In this paper, we ask two questions: is it possible to create the small labelled data set required by semi-supervised methods purely using unsupervised techniques? If so, can semi-supervised methods leverage this autonomously generated pseudo-labelled data set to deliver higher performance than state-of-the-art unsupervised approaches? We answer both these questions in the affirmative. We first find that prior approaches for identifying pseudo-labels;; perform poorly because of their low accuracy (Section 2). To create a high accuracy pseudo-labelled data set autonomously, we use a combination of ensemble of deep networks with a custom graph clustering algorithm (Section 4). We first train an ensemble of deep networks in an unsupervised manner. Each network independently clusters the input. We then compare two input data points. If all of the networks agree that these two data points belong to the same cluster, we can be reasonably sure that these data points belong to the same class. In this way, we identify all input data pairs belonging to the same class with high precision in a completely unsupervised manner. In the next step, we use these high quality input pairs to generate a similarity graph, with the data points as nodes and edges between data points which are deemed to be similar by our ensemble. From this graph, we extract tight clusters of data points, which serve as pseudo-labels. Note that, in this step, we do not cluster the entire dataset, but only a small subset on which we can get high precision. Extracting high quality clusters from this graph while ensuring that the extracted clusters correspond to different classes is challenging. We discuss our approach in Section 4.2.1 for solving this problem. In this way, our method extracts unambiguous samples belonging to each class, which serves as pseudo-labels for semi-supervised learning. For semi-supervised learning using the labels generated above, one could use ladder networks. However, we found that ladder networks is unsuitable for the initial unsupervised clustering step as it can degenerate to outputting constant values for all inputs in the absence of unsupervised loss. To enable unsupervised clustering, we augment ladder networks using information maximization to create the Ladder-IM, and with a dot product loss to create Ladder-Dot. We show in Section 5 that Ladder-IM and Ladder-Dot, by themselves, also provide improvements over previous state of the art. We use the same models for both the first unsupervised learning step as well as the subsequent pseudo-semi-supervised iterations. Finally, the approach of finding high quality clusters using an ensemble, and using them as labels to train a new ensemble of semi-supervised models, is iterated, yielding continued improvements. The large gains of our method mainly come from this iterative approach, which can in some cases, yield upto 17% gains in accuracy over the base unsupervised models (see section 5.5). We name our pseudo-semi-supervised learning approach Kingdra 1. Kingdra is independent of the type of data set; we show examples of its use on both image and text data sets in Section 5. This is in contrast to some previous approaches using CNNs, e.g. , , which are specialized for image data sets. We perform unsupervised classification using Kingdra on several standard image (MNIST, CIFAR10, STL) and text (reuters, 20news) datasets. On all these datasets, Kingdra is able to achieve higher clustering accuracy compared to current state-of-the-art deep unsupervised clustering techniques. For example, on the CIFAR10 and 20news datasets, Kingdra is able to achieve classification accuracy of 54.6% and 43.9%, respectively, delivering 8-12% absolute gains over state of the art ;. Several techniques have been proposed in the literature for generating pseudo-labels (; ; . , the output class with the highest softmax value (Argmax) is taken to be the pseudo-label. , the authors perform K-means clustering on the feature vector and use the K-means clusters as pseudo-labels. Finally, authors in treat the softmax output as confidence and only label those items whose confidence value is above a high threshold. Note that none of these techniques for identifying pseudo-labels have been applied in our context, i.e., for unsupervised clustering using semi-supervised models. In this section, we evaluate if pseudo-labels created by these prior techniques can be leveraged by semi-supervised models to improve clustering accuracy. We start with a semi-supervised model based on Ladder networks called Ladder-IM (see Section 4.1 for model details) and train using only its unsupervised loss terms on MNIST and CIFAR10 datasets. We use each of the above three pseudo-labelling approaches on the trained model to provide an initial set of pseudo-labels to the datasets (e.g., using K-means clustering on the feature vector of the model as in , etc.). We call the accuracy of these pseudo-labels the initial pseudo-label accuracy. We then use these generated pseudo-labels along with the datasets to train the model again, now with a supervised loss term (based on the pseudo-labels) and the earlier unsupervised loss terms. We again run the pseudo-labelling approaches on the newly trained model to derive an updated set of pseudo-labels. We iterate this process of training and pseudo-labelling until the pseudo-label accuracy stabilizes. We call this the final clustering accuracy. The initial pseudo-label accuracy and the final clustering accuracy for the three approaches are shown in Table 1. First, consider MNIST. The unsupervised clustering accuracy of Ladder-IM is 95.4%. Argmax simply assigns pseudo-labels based on the model's output and since this doesn't add any new information for subsequent iterations, the final accuracy remains at 95.4%. On the other hand, the pseudo-labels identified by both the K-means and threshold approaches in worse initial label accuracy (75.4% and 88.6%). When this low-accuracy pseudo-label is used as supervision to train the model further, it in a low final clustering accuracy of 60.9% and 91.6%, respectively. CIFAR10 are similar. Ladder-IM clustering accuracy is 49% which remains the same under Argmax as before. Pseudo-label accuracy using the K-means approach is worse and in pulling down the final accuracy to 44.8%. Interestingly, threshold in a slightly higher initial accuracy of 60.5% but even this is not high enough to improve the final clustering accuracy for CIFAR10. From these , we arrive at the following two . First, if the initial pseudo-label accuracy is not high, using pseudo-labels as supervision can in bringing down the final clustering accuracy. Thus, high accuracy of initial pseudo-labels is crucial for improving clustering accuracy. Second, current approaches for identifying pseudo-labels do not deliver high accuracy and hence are unable to help improve overall clustering accuracy. maximizes the mutual information between the predicted label of the image and the predicted label of the augmented image. This method uses convolution networks and requires domain knowledge of the dataset. Self-supervised learning: Another form of unsupervised learning uses auxiliary learning tasks for which labels can be self generated to generate useful representations from data. Many methods use spatial information of image patches to generate self-supervised data. E.g. predicts pixels in an image patch using surrounding patches, while predicts the relative position of image patches. uses time as a self supervisory signal between videos taken from different view points. Temporal signal is also used to learn representations from single videos by predicting future frames, e.g.. Our method uses correlation between outputs of input points across an ensemble as a supervisory signal to generate self-supervised pseudo-labels. Semi-Supervised learning: Semi-supervised approaches use sparse labelling of datapoints. propagates labels based on nearest neighbors. uses a deep version of label propagation. adjusts labels probabilities, starting with trusting only true labels and gradually increases the weight of pseudo labels. employs a denoising auto encoder architecture and have shown impressive performance. uses an averaged model over previous iterations as a teacher. Other than these, some semi-supervised learning methods like and use data augmentation and assume some domain knowledge of the dataset with some of the data augmentation specific to image datasets. and uses virtual adversarial training combined with the classification loss to perform semi-supervised classification. However, we found that these methods do not work well if we jointly train them with unsupervised losses. Ladder networks does not require any domain-dependent augmentation, works for both image and text datasets, and can be easily jointly trained with supervised and unsupervised losses. Thus, we chose to work with Ladder networks in our experiments, though our approach is general enough to work with any semi-supervised method that accommodates training with unsupervised loss terms. Unsupervised ensemble learning: Unsupervised ensemble learning has been mostly limited to generating a set of clusterings and combining them into a final clustering. Figure 1: Kingdra overview. In step 1, we train all the models using the unlabeled samples. In step 2 we construct a graph modeling pairwise agreement of the models. In step 3, we get k high confidence clusters by pruning out data-points for which the models do not agree. In step 4 we take the high confidence clusters and generate pseudo labels. In step 5 we train the models using both unlabeled samples and pseudo labeled samples. We iterate step 2 to step 5 and final clusters are generated. An overview of the Kingdra method is summarized in Figure 1. Given an unlabelled dataset X = {x 1, . . ., x n}, we start with unsupervised training of an ensemble of models M = {M 1, . . ., M m}. For the individual models, any unsupervised model can be used. However, we propose a novel Ladder-* model, in which we build on ladder networks and modify it to support clustering. Next, we use the agreement between the ensemble models on a pair of data points, as a measure of similarity between the data points. This pairwise data is used to construct a similarity graph, from which we extract k tight clusters of data points, which serve as pseudo-labels. Note that, in this step, we do not cluster the entire dataset, but only a small subset on which we can get high precision. This is important for improving the accuracy of our semi-supervised training, as discussed in section 2. These pseudo-labels then serve as training data for semi-supervised training of a new ensemble of Ladder-* models. Finally, we perform multiple iterations of the above steps for continued improvement. The first step of our method is unsupervised training of an ensemble of models. Our framework allows using any unsupervised method for this step, and we have experimented with existing approaches, such as. The accuracy of this base model directly impacts the accuracy of our final model, and thus using an accurate base model clearly helps. In that light, we have also developed a novel unsupervised model Ladder-*, which outperforms other unsupervised models in most data sets. Ladder networks have shown great success in semi-supervised setting. However, to the best of our knowledge, the ladder architecture has not been used for unsupervised clustering. One reason perhaps is that ladder networks can degenerate to outputting constant values for all inputs in the absence of a supervised loss term. To circumvent this degeneracy, we add an unsupervised loss to the regular ladder loss terms so that it directs the network to give similar outputs for similar inputs, but overall maximizes the diversity in outputs, so that dissimilar inputs are directed towards dissimilar outputs. We achieve this objective by incorporating one of two losses -the IM loss; or the dot product loss. We call the two variants Ladder-IM and Ladder-Dot, respectively. IM loss: The IM loss or the information maximization loss is simply the mutual information between the input X and output Y of the classifier: where H and H(.|.) are the entropy and conditional entropy, respectively. Maximizing the marginal entropy term H(Y), encourages the network to assign disparate classes to the inputs, and thus encourages a uniform distribution over the output classes. On the other hand, minimizing the conditional entropy encourages unambiguous class assignment for a given input. The dot product loss is defined to be which forces the network outputs for different inputs to be as orthogonal as possible. This has a similar effect to IM loss, encouraging the network to assign disparate classes to the inputs. Among Ladder-IM and Ladder-Dot, we found Ladder-IM to perform better than Ladder-Dot in most cases. However, we did find that Ladder-Dot along with Kingdra iterations outperforms when the data set has a large imbalance in the number of samples per class. The reason for this is that the dot product loss is agnostic to the number of samples per class, while the marginal entropy term in the IM loss will drive the network towards overfitting a class with more samples, compared to a class with less number of samples. A more detailed presentation of Ladder-IM and Ladder-Dot can be found in the appendix. Kingdra exploits an ensemble of Ladder-* models to further improve the performance of unsupervised learning. Note that, in supervised learning, ensembling is trivial as we can simply average the outputs of the individual models or do voting on them. On the other hand, in unsupervised learning, it is not trivial to do voting, as in the absence of training labels there is no stable class assignment for outputs across different models, and thus we do not have any mapping of class IDs of one model to another. To solve this we propose a simple approach, where we look at pairs of data-points, rather than at individual samples. Two data-points are in the same cluster with a high confidence if majority (or all) of the models in the ensembles put them in same cluster. For example, given an input pair x, x, if for enough models, we can say with high confidence that they belong to the same class. Using this pairwise approach, we propose a graph based method to find small sized, but high precision clusters. We construct a graph G = {X, E pos, E neg} with n nodes where each input data-point x is represented as a node. Here E pos and E neg are two types of edges in the graph: • Strong Positive Edges: A strong positive edge is added between two data-points when a large number of models agree on their predicted class. (x, x) ∈ E pos ⇐⇒ n_agree(x, x) ≥ t pos where t pos is a chosen threshold, and n_agree(x, x) = |{m : m ∈ M, m(x) = m(x)}|. • Strong Negative edges: A strong negative edge is added between two data-points when a large number of models disagree on their predicted class. (x, x) ∈ E neg ⇐⇒ n_disagree(x, x) ≥ t neg, where t neg is a chosen threshold, and n_disagree(x, x) = |{m : m ∈ M, m(x) = m(x)}|. A strong positive edge between two data points, implies that most models believe they are in the same class, while a strong negative edge between two data points implies that most models believe they should belong to different classes. Algorithm 1 Get high precision clusters using ensembles 1: procedure GETCLUSTERS(X, k) 2: for k ∈ {1, 2, . . ., k} do 4: x max = argmax x∈X {|(x, x) ∈ E pos |} 5: for x ∈ X do 7: end for 9: end for 10: Return S = {S 1, S 2, . . ., S k} 11: end procedure After building the graph, each clique of strong positive edges would be a cluster, where within a clique, data-points belong to the same class with high confidence. Since we add only high confidence edges to the graph, the number of cliques can be much larger than k. Hence we need to select k cliques where we would like to maximize the size of each clique, but also require that the cliques are diverse (in order to not select two cliques with data-points belonging to the same class). Hence, within a clique, nodes should be connected by strong positive edges, while across cliques, nodes should be connected by strong negative edges. As finding cliques is not solvable in polynomial time, we use a simple and efficient greedy approximation algorithm, as shown in Algorithm 1. Rather than finding cliques, we greedily find nodes with the highest number of strong positive edges (line 4). The intuition is that most of the neighbours of that node will also be connected with each other. In the case of Cifar-10, we find that with a threshold of 90%, 81% of nodes are fully connected with each other. If the threshold is 100%, all nodes in a cluster are connected with each other by transitivity. We take the node with highest number of strong positive edges, along with other nodes connected to it by strong positive edges and add them to a cluster (line 5). We then remove all the nodes that do not have a strong negative edge to the chosen node (line 6-7). The intuition here is that these nodes are not diverse enough from the chosen cluster (since some models think that they belong to the same class as the currently chosen node), and thus should not be part of the next set of chosen clusters. By repeating the process k times, we get k diverse clusters, approximately satisfying our requirement. Once the high precision clusters are identified, we treat these clustered points (points in set S) as pseudo-labels, and solve our unsupervised clustering problem using a semi-supervised method. Although any semi-supervised method can be used, as described in section 4.1 we use the proposed Ladder-* method, which we found superior to ladder networks in our experiments. Instead of training a single semi-supervised model, we train an ensemble of models, and again use them to find high quality clusters. This approach can be iterated, yielding continued improvements. We name this approach Kingdra. Algorithm 2 describes the complete Kingdra algorithm. First, the individual models are trained using only the unsupervised Ladder-* loss (lines 1-4). Then, for each of the iterations, we obtain high precision clusters (line 6), derive pseudo-labels from them (line 8), and then train the models with both the unsupervised and supervised losses (lines 9-10). We compute the pseudo-labels using the mini-clusters as follows. For a model M j ∈ M and clusters S, we need to find an appropriate mapping of the clusters to the output classes of the model. In particular, for a cluster S ∈ S, we assign all data-points in S the following label: That is, we map a cluster to the output class to which most data-points in the cluster are mapped. These pseudo-labels are then used for computing the supervised loss of Ladder-*. This iterative approach leads to a continuous improvement of clustering quality. We observe that the size of clusters returned by Algorithm 1 increases after each iteration until they cover almost the entire input set. The clustering performance of the model also generally improves with each iteration until it saturates, as we show in Section 5. We also note that cluster assignments become more stable with subsequent end for 12: end for 13: Use averaging on the ensemble models M to return final clusters iterations, which also leads to decrease in variance across multiple runs. That is, the variance across multiple runs decreases if we run Kingdra for more iterations. In this section we evaluate the performance of Kingdra on several popular datasets. For a fair comparison, we use the same data pre-processing and same model layer sizes as prior work. We evaluate Kingdra on three image datasets and two text datasets: MNIST is a dataset of 70000 handwritten digits of 28-by-28 pixel size. Here, the raw pixel values are normalized to a range 0-1 and flattened to vector of 784 dimensions. CIFAR10 is a dataset of 32-by-32 color images with 10 classes having 6000 examples each. STL is a dataset of 96-by-96 color images with 10 classes having 1300 examples each. For CIFAR10 and STL raw pixels are not suited for our goal as the color information dominates, hence as mentioned in , we use features extracted from a Resnet-50 network pre-trained on the ImageNet dataset. Reuters is a dataset containing English news stories with imbalanced data and four categories. We used the same pre-processing as used by; after removing the stop-words, tf-idf features were used. 20News is a dataset containing newsgroup documents with 20 different newsgroups. Similar to , we remove stop words and keep 2000 most frequent words, and used tf-idf features. All our experiments were performed using the same pre-processed data. We use standard unsupervised evaluation methodology and protocol to compare different methods. , we set the number of clusters the same as the number of ground truth classes and evaluated unsupervised clustering accuracy as: where l i and c i are the ground truth cluster label and the cluster label assigned by the model respectively. We find the best one-to-one mapping of ground truth label and model generated clusters with p ranging over all one-to-one mappings. 89.6 (5.4) 92.8 (2.5) 45.5 (2.9) 71.9 (6.5) 24.4 (4.7) IMSAT (VAT) 98 We compare Kingdra against several clustering algorithms on our datasets. Specifically, we compare against traditional clustering algorithms such as K-Means and Agglomerative clustering(AC). We also compare against representation learning baselines where we use models such as Deep Autoencoders(dAE), Deep Variational Auto-encoders (dVAE), and then use K-Means on the learned representations. Finally, we also compare our model with deep learning based clustering methods such as Deep RIM, DEC, DeepCluster, and IMSAT. Deep RIM uses a multi-layer neural network with the RIM objective. DEC iteratively learns a lower dimensional feature representation and optimizes a clustering objective. We also compare with two versions of IMSAT -IMSAT(RPT) and IMSAT(VAT) where data augmentation is used to impose invariance in the model outputs. For a fair comparison the network architecture we used for DeepCluster is same as other models. For our , we report the performance of Ladder-IM and Ladder-Dot individually, and finally Kingdra that includes an ensemble of Ladder-* networks, along with the semi-supervised iterations. We used Tensorflow and Keras for our implementation. For the model architecture, we use two fully connected layers of 1200 neurons each with RELU activation followed by a final layer of neurons corresponding to number of classes, similar to. All hyper-parameters were kept the same when evaluating across the five data sets. The standard deviation of noise, required by ladder, was based on the L2 distance of data points in each data set and all other hyper-parameters were based on defaults used in. We used Adam as the optimizer with the default learning rate. All experiments used ten models as part of the ensemble. For the graph clustering algorithm, we set t pos and t neg to be the number of models (ten). Accuracy of prior approaches and ours is shown in Table 2. As can be seen from the table, Ladder-IM by itself delivers good performance and Kingdra-Ladder-IM achieves higher clustering accuracy than state-of-the-art deep unsupervised approaches such as and in all five data sets. Further, the gap between Kingdra and prior approaches is significant in two data sets: Kingdra-Ladder-IM achieves an average accuracy of 54.6% for CIFAR10 compared to 45.6% for IMSAT and 46.9% for DEC -an 8% increase in absolute accuracy. Similarly, Kingdra-Ladder-IM achieves an average accuracy of 43.9% for 20news compared to 31.1% for IMSAT and 30.8% for DEC -an increase of over 12% in absolute accuracy. Note that while deep networks are state-of-the-art for most data sets, linear approaches outperform deep approaches on 20news with linear RIM achieving 50.9% accuracy. We also tried DeepCluster Caron An interesting aspect to note is that the use of an ensemble by itself only provides small gains of 1-2%, similar to what one expects from ensembles in supervised learning (e.g., compare Ladder-IM with Ladder-IM-ensemble). The large gains mainly come from Kingdra using the ensemble to generate pseudo-labels, which is then iterated. For example, Kingdra-Ladder-IM provides absolute gains of 4-6% in most data sets over the base model. Similarly, Kingdra-Ladder-Dot provides absolute gains of 9% in MNIST and 17% in STL over the base Ladder-Dot model. Thus, our approach of generating pseudo-labels from ensembles is a powerful approach that delivers large gains in unsupervised learning. Also note that Kingdra-Ladder-IM performs better than Kingdra-Ladder-Dot for most data sets except for the Reuters data set where the latter performs better (Reuters has a large class imbalance with the largest class representing 43% of the data). Finally, note the standard deviation of the various approaches shown in the Table. One can see that Kingdra in general in lower standard deviation than many of the prior approaches even while delivering higher accuracy. Figure 2 shows the accuracy of pseudo-labels and Kingdra-Ladder-IM, as well as number of pseudolabels identified by the graph clustering algorithm vs the number of iterations for STL, CIFAR10, and MNIST datasets. As iterations progress, the accuracy of pseudo labels decreases as more pseudolabels get added; however, this still helps improve the overall clustering accuracy. Note that, unlike pure semi-supervised approaches which use a small set of (randomly sampled) data points that match the input data distribution, our pseudo-labels do not completely match the input data distribution (since our selection algorithm is biased towards easy data points). This causes an increased gap between the accuracy of pseudo-labels, and that of overall clustering. Finally, we include qualitative analysis of our experimental in the appendix. In this paper, we introduced Kingdra, a novel pseudo-semi-supervised learning approach for clustering. Kingdra outperforms current state-of-the-art unsupervised deep learning based approaches, with 8-12% gains in absolute accuracy for CIFAR10 and 20news datasets. As part of Kingdra, we proposed clustering ladder networks, Ladder-IM and Ladder-Dot, that works well in both unsupervised and semi-supervised settings. The Regularized Information Maximization (RIM) approach for unsupervised learning was introduced in and extended by for multi-dimensional setting. The RIM method minimizes the following objective for a classifier: where R(θ) is a regularization term, and I(X; Y) is the mutual information between the input X and output Y of the classifier. The mutual information can be written as the difference between marginal entropy and conditional entropy: where H and H(.|.) are entropy and conditional entropy, respectively. Maximizing the marginal entropy term H(Y), encourages the network to assign disparate classes to the inputs, and thus encourages a uniform distribution over the output classes. On the other hand, minimizing the conditional entropy encourages unambiguous class assignment for a given input. In the unsupervised setting, where other priors are not known, this loss makes intuitive sense. For the regularization loss term R(θ) above, many options have been proposed. , for example, propose a Self-Augmented Training (SAT) loss, which imposes invariance on the outputs of original and slightly perturbed input data. The authors experimented with random perturbation (IMSAT-RPT), and adversarial perturbation (IMSAT-VAT) where the perturbation is chosen to maximize the divergence between the two outputs on the current model. Ladder networks have shown impressive performance for semi-supervised classification. They employ a deep denoising auto encoder architecture, in which an additive noise is added to each hidden layer in the encoder, and the decoder learns a denoising function for each layer. The objective function is a weighted sum of supervised cross entropy loss on the output of the noisy encoder, and a squared error of the unsupervised denoising loss for all layers. Unlike standard auto-encoders, ladder networks also add lateral skip connections from each layer of the noisy encoder to the corresponding decoder layer. The additive noise acts as a regularizer for the supervised loss, while the lateral connections in the denoising decoder layers enable the higher layer features to focus on more abstract and task-specific features. for a detailed analysis. Borrowing the formalism in , a ladder network with L encoder/decoder layers can be defined as: where θ j and φ j are the parameters for the Encoder and Decoder, respectively. The variables z are the hidden layer outputs for the clean, noisy, and denoised versions at layer k, respectively. x, y i,ỹ i are the input, clean output and the noisy output, respectively. The objective function consists of the reconstruction loss between clean and decoded intermediate features: and a supervised cross entropy loss on the output of the noisy encoder (which is used only in the semi-supervised setting): We now describe our novel Ladder-IM and Ladder-Dot models. The unsupervised denoising loss in Equation 7, along with the lateral connections architecture enables ladder networks to learn useful features from unsupervised data. However, in the absence of any supervised loss (Equation 8), ladder networks can degenerate to the trivial solution of a constant output for each encoder layer, as the decoder can then simply memorize these constants to make the denoising loss zero. Having batch normalization layers helps to alleviate this problem, but the loss function still allows the trivial solution. On the other hand, the mutual information loss (Equation 6) in RIM methods, in particular the marginal entropy term H(Y), encourages the network to assign disparate classes to the inputs. Ladder-IM: Combining ladder networks with information maximization can fix the above degeneracy problem, while simultaneously encouraging the ladder output towards a uniform distribution. We use both the clean, and noisy outputs of the ladder network for computing the mutual information loss, i.e. where Y = {y 1, . . ., y N} is the set of clean outputs, andỸ = {ỹ 1, . . .,ỹ N} is the set of noisy outputs from the ladder network. Another way of thinking about the Ladder-IM approach is completely within the RIM framework. The unsupervised ladder loss loss denoise, can be simply thought of as the regularization term R(θ) in equation 5. To that effect, we also add another regularization loss term, which is the KL divergence between the clean and noisy outputs of the ladder network encoder, i.e. This regularization can be thought of as a generalization of the random perturbation loss proposed in , where the authors impose invariance on the outputs of original and randomly perturbed inputs. Our regularization based on adding noise to the hidden layers is similar to dropout , and can be thought of as adding higher level feature noise, rather than just input noise. Thus, in the unsupervised case, this would lead to the following minimization objective: In this paper, we set α and β to one. Finally, in the semi-supervised case, we also add the supervised cross entropy term (Equation 8), as done in the original ladder networks. Ladder-Dot: We also try a dot product loss to fix the above degeneracy problem. The dot product loss is defined to be which forces the network outputs for different inputs to be as orthogonal as possible. This has a similar effect to IM loss, encouraging the network to assign disparate classes to the inputs. Among Ladder-IM and Ladder-Dot, we found Ladder-IM to perform better than Ladder-Dot in most cases. However, we did find that Ladder-Dot along with Kingdra iterations outperforms when the data set has a large imbalance in the number of samples per class. The reason for this is that the dot product loss is agnostic to the number of samples per class, while the marginal entropy term in the IM loss will drive the network towards overfitting a class with more samples, compared to a class with less number of samples. Overall, we found in our experiments that Ladder-IM showed superior performance to on most data sets. Moreover, in pure semi-supervised settings also, Ladder-IM outperformed vanilla ladder networks in our preliminary analysis. C QUALITATIVE ANALYSIS Figure 3 shows the similarity graph obtained after the first three iterations of Kingdra on the MNIST dataset. As the iteration progresses, one can see that there are fewer inter-cluster linkages indicating that the models are converging on the labels for these data points. Figure 4 shows randomly selected examples from our final clusters generated by Kingdra. One can see that the examples are highly accurate for MNIST, thus ing in high overall accuracy. However, for CIFAR10, there are several incorrectly labelled examples, including two clusters which do not have a clear mapping with any ground truth class, thereby ing in much lower overall accuracy. We evaluated the accuracy of KINGDRA-LADDER-IM as the number of models in the ensemble was varied. MNIST accuracy with 1, 2, 5, 10, and 15 models is 95.0, 96.2, 97.4, 98.5, and 98.5 respectively. This suggests that accuracy saturates after 10 models and we use 10 models for our ensemble for all our experiments. We have an efficient implementation of clustering, which takes 210s for largest n = 70000. On a server with four P100 GPUs, CLadder-IM takes 2mins, CLadder-IM with ensemble takes 8mins and Kingdra with 10 iterations takes 80mins while IMSAT(RPT) takes 5mins. Here we give an analysis of , explaining the shortcomings. We observed that the clustering accuracy generally decreases with iterations. This is because the pseudolabels generated could be bad, which in worse accuracy in the next iteration. On the other hand, our approach only uses small number high-confidence samples for pseudo-labels. While Kingdra performs well in the datasets we studied, the similarity-based graph clustering algorithm used has difficulty as the number of classes increase. For example, for the datasets we evaluated, the t pos and t neg can be simply set to the number of models in the ensemble. However, as the number of classes increase, these thresholds may need some tuning. For CIFAR100, with 100 classes, our graph clustering algorithm is not able to identify 100 diverse classes effectively. We are looking at improving the clustering algorithm as part of future work. We are also evaluating adding diversity to the models in the ensemble, either via changing the model structure, size and/or through changing the standard deviation of random noise used in ladder networks. • MNIST: A dataset of 70000 handwritten digits of 28-by-28 pixel size. The raw pixel values are normalized to a range 0-1 and flattened to vector of 784 dimensions. • STL: A dataset of 96-by-96 color images with 10 classes having 1300 examples each. We do not use the 100000 unlabeled images provided in the dataset. Similar to ], features are extracted using 50-layer pre-trained deep residual networks. • Reuters: A dataset containing English news stories with four categories: corporate/industrial, government/social, markets, and economics. We used the same preprocessing as used by. After removing the stop-words, td-idf features were used. • 20News: A dataset containing newsgroup documents with 20 different newsgroups. Similar to after removing stop words and keeping 2000 most frequent words, td-idf features were used.

Represent the TLDR for finding the corresponding scientific article

Using ensembles and pseudo labels for unsupervised clustering

Represent the short summary for finding the corresponding scientific article (long)

Our proposed algorithm does not use all of the unlabeled data for the training, and it rather uses them selectively.