Question:
represent the text to find a very short summary (1-2 sentences max)
E.g. 'Generative Adversarial Networks (GANs) have shown impressive in modeling distributions over complicated manifolds such as those of natural images. However, GANs often suffer from mode collapse, which means they are prone to characterize only a single or a few modes of the data distribution. In order to address this problem, we propose a novel framework called LDMGAN. We ﬁrst introduce Latent Distribution Matching (LDM) constraint which regularizes the generator by aligning distribution of generated samples with that of real samples in latent space. To make use of such latent space, we propose a regularized AutoEncoder (AE) that maps the data distribution to prior distribution in encoded space. Extensive experiments on synthetic data and real world datasets show that our proposed framework signiﬁcantly improves GAN’s stability and diversity. Generative models (; ; ; ; ; ;) provide powerful tools for unsupervised learning of probability distributions over difficult manifolds such as those of natural images. Among these models, instead of requiring explicit parametric specification of the model distribution and a likelihood function, Generative Adversarial Networks (GAN) only have a generating procedure. They generate samples that are sharp and compelling, which have gained great successes on image generation tasks (; ; ;) recently. GANs are composed of two types of deep neural networks that compete with each other: a generator and a discriminator. The generator tries to map noise sampled from simple prior distribution which is usually a multivariate gaussian to data space with the aim of fooling the discriminator, while the discriminator learns to determine whether a sample comes from the real dataset or generated samples. In practice, however, GANs are fragile and in general notoriously hard to train. On the one hand, they are sensitive to architectures and hyper-parameters . For example, the imbalance between discriminator and generator capacities often leads to convergence issues. On the other hand, there is a common failure issue in GANs called mode collapse. The generator tends to produce only a single sample or a few very similar samples in that case, which means GANs put large volumes of probability mass onto a few modes. We conjecture the mode missing issue in GANs is probably because GANs lack a regularization term that can lead the generator to produce diverse samples. To remedy this problem, in this work, we first propose a regularization constraint called Latent Distribution Matching. It suppresses the mode collapse issue in GANs by aligning the distributions between true data and generated data in encoded space. To obtain such encoded space, we introduce a regularized autoencoder which maps data distribution to a simple prior distribution, eg., a gaussian. As shown in Figure 1, we collapse the decoder of the regularized AE and generator of GAN into one and propose LDMGAN. Our framework can stabilize GAN's training and reduce mode collapse issue in GANs. Compared to other AE-based methods on 2D synthetic, MNIST, Stacked-MNIST, CIFAR-10 and CelebA datasets, our method obtains better stability, diversity and competitive standard scores. GANs were initially proposed by. They contain two neural networks: a generator and a discriminator. Let {x i} N i=1 denote the training data, where each x i ∈ R D is drawn from unknown data distribution p d (x). Generator is a neural network G γ that maps the noise vector z ∈ R K, typically drawn from a multi-variate gaussian, to data spacex ∈ R D. The discriminator D ω is a classification network that distinguishes real samples from generated samples. The parameters of these networks are optimized by solving the following minmax game: where E x indicates an expectation over the data distribution p d (x) and E z indicates an expectation over the prior noise distribution p(z). Given generator G, the optimal discriminator D is is the model distribution. With the help of the optimal discriminator, the GAN objective is actually minimizing the Jensen Shannon divergence between the model distribution and the data distribution. And the global minimum is achieved if and only if There are many works that try to stabilize GAN's training and alleviate mode collapse in GANs. DCGAN designed a class of deep convolutional architectures which has become standard architecture for training GANs. Improved GANs proposed several techniques like feature matching, mini-batch discrimination and historical averaging which stabilized GAN's training and reduced mode collapse. Unrolled GAN proposed unrolling optimization of the discriminator objective to train generator. TTUR introduced two time-scale update rule and proved its convergence to a local Nash equilibrium. analyzed the convergence properties of GANs and proposed WGAN which leveraged the Wasserstein distance and demonstrated its better convergence than Jensen Shannon divergence. However, WGAN required that the discriminator must lie on the space of 1-Lipschitz functions. It used a weight clipping trick to enforce that constraint. WGAN-GP stabilized WGAN by alternating the weight clipping by penalizing the gradient norm of the interpolated samples. SN-GAN proposed spectral normalization which controls the Lipschitz constraint of netowork layers. Some works make efforts towards integrating AEs into GANs. AEs can be used in discriminators. EBGAN employed an autoencoder structure in its discriminator and introduced repelling regularization to prevent mode collapse. BEGAN extended EBGAN by optimizing Wasserstein distance between AE loss distributions of real and generated samples. MDGAN contained a manifold step and a diffusion step that combined a plain autoencoder with GANs to suppress mode missing problem. However, MDGAN required two discriminators and it did not impose regularization on its AE. VAEGAN unified VAE and GAN into one model, and utilized feature-wise distance in intermediate features learned by discriminator to replace similarity metric in data space in VAE to avoid blur. AAE used adversarial learning in its encoded space to perform variational inference. BiGAN and Adversarial Learned Inference (ALI) jointly trained an inference model and a generative model through adversarial game. VEEGAN employed a reconstructor network to both map the true data distribution p d (x) to a gaussian and to approximately invert the generator network. AGE performed adversarial game between a generator and an encoder, and proposed to align the model distribution with data distribution in encoded space. They eliminated discriminator in their framework and used an encoder to extract statistics to align model distribution with data distribution. The most related works to ours are VAEGAN and VEEGAN. VAEGAN's motivation was to combine VAE with GAN, and they relied on an ELBO, namely While we perform marginal distribution matching, namely E q(z|x) [log p(x|z)] − D KL (q(z) p(z)), and view the second term as a regularization term. Therefore there is no need of re-parameterization trick in our framework. Also we compare the distributions of the real samples and generated samples in the encoded space to suppress mode collapse in GANs which is different from VAEGAN. VEEGAN had a reconstructor network which mapped the data distribution to a gaussian as well. However, VEEGAN autoencoded noise vectors rather than data items, and its training objective was formed as an adversarial game in joint space (x, z) like BiGAN and ALI. Our regularized autoencoder is similar to AAE. But instead of using adversarial learning in encoded space which also has possibility of falling into mode collapse as in data space, we use explicit divergence to align aggregated posterior with the prior distribution. GAN's training can be viewed as an adversarial game between two players, in which the discriminator tries to distinguish between real and generated samples, while the generator tries to fool the discriminator by pushing the generated samples towards the direction of higher discrimination values. However, there are many points in data manifold that have high discrimination values. When collapse to a single mode is imminent, the gradient of the discriminator may point in similar directions for many similar points . Since the discriminator iteself processes each sample independently, there is no mechanism to tell the outputs of the generator to become more dissimilar to each other. In this paper, we propose a framework called LDMGAN to address this problem. See Figure 1 for a glance of our model's architecture. Instead of exposing the dissimilarity of generated samples to the discriminator , we introduce a regularized autoencoder in our framework. We perform data dimensionality reduction on the training data to obtain a well-shaped manifold in encoded space, e.g., a factorized gaussian, by learning a regularized autoencoder. Then when sampling noise from a factorized gaussian to feed into the generator to produce generated samples, we encode the generated samples to latent space to enforce them to be coincident with the factorized gaussian. In this way, the generator receives a signal from that constraint to tell it to produce diverse samples. We call this constraint Latent Distribution Matching (LDM). In the following, we describe LDM and the regularized autoencoder in detail. Figure 1: Our LDMGAN's architecture. The thick lines denote added losses to GANs. The main idea of LDM is to introduce an encoder function F θ that can map the true data distribution p(x) to a prior Gaussian p(z). To understand why such an encoder helps alleviate address mode collapse, consider the example in Figure 2. The middle vertical panel represents the data space, where in this example the true data distribution p d (x) is a mixture of two Gaussians. The bottom panel depicts the input to the generator (latent code space), which is drawn from p(z), and the top panel depicts the of applying the encoder network to the generated data and true data. Assume that the encoder approximately maps the data distribution to the prior gaussian (see q(z) above the top panel in Figure 2 ). The purple arrows above the middle panel show the action of the encoder on the true data, whereas the green arrows show the same action on the generated data. If mode collapse occurs as it does in this example that the generator G γ captures only one of the two modes of p d (x), it would cause distribution mismatch between the two marginal distributions q(z) and q (z). The definitions of q(z) and q (z) go as follow: where. This mismatch is evaluated by a divergence measure ∆(q (z) q(z)) in the latent space. We only require this divergence to be nonnegative and zero if and only if the distributions are identical (∆(q (z) q(z)) = 0 ⇐⇒ q (z) = q(z)). Therefore, this mismatch value function is defined as: However, comparing two empirical (hence non-parametric) distributions q (z) and q(z) is difficult. We avoid this issue by introducing an intermediate reference distribution which is also the prior distribution p(z), ing in: As we have assumed that the encoder network can approximately map the data distribution to a gaussian, we can get rid of ∆(q(z) p(z)) with minor error. Finally, this mismatch term turns out to be It can be used to detect mode collapse in generator. We call this mismatch term as Latent Distribution Matching (LDM), and use this term to regularize the generator in GANs to combat mode collapse. AutoEncoders have long been used for data dimensionality reduction and feature learning. We impose the regularization that the distribution in the encoded space of real data is coincident with the prior distribution p(z) on the autoencoder. We call this autoencoder the regularized autoencoer in this paper. There are two ways of utilizing the regularized autoencoder. One is to pretrain a regularized autoencoder, then use the encoder of the regularized autoencoder only. The other is to combine the autoencoder training with GAN objective which has already been proposed in many GAN variants (; ;). We combine autoencoder with GANs in our model for three reasons: to guarantee the learning is grounded on all training samples; to regularize the generator in producing samples to resemble real ones; last but not least, to obtain an encoder that can approximately map the data distribution to a gaussian. In the following, we will explain the reasons for combining autoencoders in our model and describe our framework formally. One of the reasons that mode collapse occurs is probably that the areas near the missing modes are rarely visited by the generator, therefore providing very few samples to improve the generator around those areas. And only when G(z) is very close to the missing modes can the generator get gradients to push itself towards the missing modes. Consider a model combining GANs with reconstruction loss in data space d(x, G(E(x))). Given enough data examples x available in training data for minor mode M 0, we know G • E is a good autoencoder, then G • E(x) will be located very close to the minor mode M 0. Therefore the gradient from the discriminator ∇ log D(G • E(x)) might be able to push these points towards the minor mode M 0 and the learning is grounded on all training samples. For that purpose, the following loss is added to the generator: As the regularized autoencoder combined with GAN has imposed gaussian on the encoded space of real data samples, this guarantees such E(x) for minor modes can be sampled with fair possibility when sampling z from gaussian that are feeded to the generator. In addition, as lots of image synthesis works ) have used L 1 or L 2 loss in data space and achieved successes in producing realistic images. We use the autoencoder loss d(x, G • E(x)) in data space as well for the purpose of regularizing the generator in producing samples to resemble real ones. In this section, we formally demonstrate the regularized autoencoder we combine with GANs. Let x be the data sample vector,x be the reconstructed sample vector of x by the autoencoder and z be the latent code vector of the autoencoder. Let p(z) be the prior distribution we want to impose on the codes, q(z|x) be an encoding distribution and p(x|z) be the decoding distribution. Also let p d (x) be the data distribution, and p g (x) be the model distribution. We desire an autoencoder whose aggregated posterior q(z) is coincident with a prior distribution p(z). In this paper, we use a factorized gaussian as the prior distribution. The encoding function of the autoencoder q(z|x) defines the aggregated posterior in Equation 2. We choose a deterministic function form for q(z|x) which means the stochasticity of q(z) comes only from the the data distribution p d (x). We impose a regularization term on the encoded space of the autoencoder to minimize: ∆(q(z) p(z)). The autoencoder, meanwhile, attempts to minimize the reconstruction error d(x,x) (here d is some similarity metric, besides L 1 or L 2 distance in data space, it can also be distance metric in the feature-wise space like the intermediate layers of well-known VGG network or the discriminator of GANs). Therefore,the regularized autoencoder is trained in two phases: the reconstruction phase and the regularization phase. In the reconstruction phase, the regularized autoencoder uses SGD to update the encoder and decoder's parameters to reconstruct the input data vector. In the regularization phase, the autoencoder uses SGD to update the encoder's parameters to match the aggregated posterior with the prior distribution. Implemented by deep neural networks, with arbitrary capacity, the universal approximation theorem guarantees the regularized autoencoder is capable of approximately matching the aggregated posterior q(z) with the prior distribution p(z). Given the encoder E θ which is capable of approximately mapping the data distribution to a gaussian, consider a standard GAN training loss plus the LDM (Latent Distribution Matching) constraint on the generator: This training objective Equation 8 is what we apply in GAN framework when not concerning combining the regularized autoencoder into GAN. And after unifying the regularized autoencoder and GAN described above, the objective becomes: where λ 1 and λ 2 are hyper-parameters that control the loss weights of reconstruction error and LDM constraint respectively. Note that the LDM term ∆(q (z) p(z)) is minimized only with respect to G γ, which means the parameters of the encoder E are fixed during backpropagating the LDM term. The divergence measure between the empirical distribution and the prior distribution in the latent M -dimensional space that we use in this paper goes as follow: where µ i and σ i are the means and the standard deviations of the fitted empirical distribution Q along various dimensions. The training algorithm and architecture of our proposed model is depicted in Algorithm 1 and Figure 1 respectively. Algorithm 1 Our LDM-GAN's training algorithm 1: Dω, E θ, Gγ ← initialize network parameters respectively 2: repeat 3: X ← Random mini-batch of m data points from dataset 4: Xrec ← Gγ(Z) 8: Zrec ← E θ (X) 9: // Training discriminator D 10: // Training encoder E 12: // Training generator G 14: gγ ← ∇γ log(1 − Dω(X)) + log(1 − Dω(Xrec)) + λ1d(X, Xrec) + λ2DKL(q (Zrec) p(z)) 15: // Using SGD to update model's parameters 16: ω ← ω − ηgω; θ ← θ − ηg θ; γ ← γ − ηgγ 17: until convergence 18: return D, E, G In this section, we compare our model with four different models on 2D Synthetic, MNIST, Stacked-MNSIT, CIFAR-10 and CelebA datasets. We do not finetune the hyper-parameters λ 1 and λ 2 in our model, and by default set λ 1 = 1 and λ 2 = 1. Evaluation of mode collapse in GANs is very difficult especially when trained on natural images. However, the missing modes can be calculated precisely when using synthetic data for training. In this section we compare five different models on two synthetic datasets: a mixture of eight 2D Gaussian distributions arranged in a ring, and a mixture of twenty-five 2D Gaussian distributions arranged in a grid. The covariance matrices and centroids have been chosen such that the distributions exhibit lots of modes separated by large low-probability regions, which make them decently hard tasks despite the 2D nature of the dataset. All models share the same network architectures for fair comparison. The encoder and the generator network are consisted of two hidden layers, while the discriminator network only has one hidden layer. All networks are implemented by fully-connected layers and each layer has 64 neurons. The dimension of the prior input is 2. For VEEGAN and BiGAN, the discriminator's input is augumented by the size of the latent code. Note that the VAEGAN we used here is slightly different from the original paper, since the network architectures are shallow and the data is low-dimensional: we used L 2 loss on the data space for simplicity instead of on features learned by the discriminator. To quantify the mode collapse behavior we report two metrics: the number of modes that each model captures and the percentage of high quality samples. We draw 2048 samples from each model for evaluation. The high quality samples are defined as the points which are within three standard deviations of the nearest mode, and if there are more than twenty such samples registered to a mode, the mode is considered as captured. The are depicted in Table 1 and the numbers are averaged over eight runs. We also show the generated distribution of each model in Figure 3 As we can see from Table 1, our model captures the greatest number of modes on both synthetic datasets. The generator learned by our model is sharper and closer to the true distribution as shown in Figure 3. Though other models may obtain greater percentage of high quality samples, for example, vanilla GAN generates 99.96% high quality samples on 2D Ring dataset. However, it runs into severe mode missing problem. Vanilla GAN only captures one mode on 2D Ring dataset. 1. VEEGAN is able to capture most of the modes in some settings (LDMGAN also performs as well as VEEGAN under those settings), but it fails to generalize well under our settings. In particular, while our model resembles VAEGAN in some ways, VAEGAN suffers from some degree of mode missing problem as well and it is far from competitive compared to our proposed model. In order to systemically explore the effect of our proposed LDMGAN on alleviating mode collapse in GANs, we use a large scale grid search of different GAN hyper-parameters on the MNIST dataset. We use the same hyper-parameters settings for both vanilla GAN and our LDMGAN, and the encoder in our model is the "inverse" of the generator. The search range is listed in Table 2. Our grid search is similar to those proposed in;. Please refer to it for detailed explanations regarding these hyper-parameters. In order to estimate both the missing modes and the sample qualities, we first train a regular CNN classifier on the MNIST digits, and then apply it to compute the MODE Score which is proposed in. MODE Score is a variant of Inception Score (IS) and its definition goes as follow: where p(y|x) is the softmax output of a trained classifier of the labels, and p * (y) and p(y) are the marginal distributions of the label of the generated samples and training samples respectively. We train each architecture for 50 epochs and after that draw 10K samples for evaluation. The ing distribution of MODE Score is shown in Figure 4. It can be seen that our model clearly improves the sample quality and diversity compared to GANs. , we evaluate our method on the stacked MNIST dataset, a variant of the MNIST data specifically designed to increase the number of discrete modes. The data is synthesized by stacking three randomly sampled MNIST digits along the color channel ing in a 28×28×3 image. We now expect 1000 modes on this dataset, corresponding to the number of possible triples of digits. We generate a dataset of 50000 images for training. And for fair comparation we use the same architecture for all models. The generator and discriminator networks are implemented following DCGAN. For encoder network, we use a simple two layer MLP network without any regularization layers. For BiGAN and VEEGAN, we concatenate the last convolutional layer of the discriminator with the latent code which follows by two fully-connected layers in order to perform joint space discrimination. The dimension of the prior input is 64. Each model is trained for 50 epochs. As the true locations of the modes in this data are unknown, the number of modes are estimated using a trained classifier as described in subsection 5.2. We draw 26000 samples from each model for evaluation. As a measure of quality, following , we also report the KL divergence between the model distribution and the data distribution. The are averaged over 5 runs. As reported in Table 3, our model recovers greatest number of modes and better matches the data distribution than any other methods. Table 3: Degree of mode collapse, measured by modes captured and sample quality (as measured by KL) on Stacked-MNIST. Our model captures the most modes and also achieves the highest quality. For CIFAR-10 and CelebA datasets, we measure the performance with FID scores . FID can detect intra-class mode dropping, and measure the diversity as well as the quality of generated samples. FID is computed from 10K generated sampels and the pre-calculated statistics on all training data provided in the code repository 2. Our default parameters are used for all experiments. And the model architecture is the same as the standard DCGAN. We train each model for 50 epochs on CIFAR-10 and 30 epochs on CelebA. The dimension of the prior input is 100. All of our experiments are conducted using the unsupervised setting. We evaluate the FID score for each model per epoch during training, and leave the best one per run. As we can not reproduce VEEGAN and BiGAN on these datasets using the same architecture, we do not report FID scores on these two models. The reported are depicted in Table 4 and are averaged over 5 runs. It is important to mention that we do not pursue the state-of-the-art FID scores on both datasets as we only want to prove that our model improves GANs. It can be seen that our model has both the best FID scores and averaged FID scores on CIFAR-10 and CelebA. The improvement on CelebA is smaller compared to that on CIFAR-10. We suspect that this is due to the less complicated structure of face images, but anyway our model still consistently performs better than DCGAN and VAEGAN. The random samples generated from different models on both datasets are shown in Appendix A. Finally, to show that our proposed model are not just simply memorizing the training data, we interpolate between random vectors on generator of our model trained on CelebA dataset and the are depicted in Appendix A. CelebA Table 4: FID scores for different models, lower is better. Best FID is the best run in 5 trial. We propose a robust AE-based GAN model with novel Latent Distribution Matching constraint, called LDM-GAN, that can address the mode collapse effectively. Our model is different from previous works: We propose a novel regularization constraint called Latent Distribution Matching to align the distributions of real data and generated data in encoded space. We propose a new regularized autoencoder for approximately mapping data distribution to a gaussian. We designed a novel AE-based GAN that drastically alleviate mode collapse in GANs. Extensive experiments demonstrate that our method is stable and does not suffer from mode collapse problem in Synthetic 2D, MNIST, Stacked-MNIST datasets. Furthermore, we achieve better FID scores on CIFAR-10 and CelebA datasets compared to baseline models without any other tricks and techniques. These demonstrate that our proposed LDMGAN indeed improves GAN's stability and diversity. A APPENDIX' == 'We propose an AE-based GAN that alleviates mode collapse in GANs.' != 'We propose a method based on the adversarial training strategy to learn discriminative features unbiased and invariant to the confounder(s) by incorporating a loss function that encourages a vanished correlation between the bias and learned features.'

Question: Despite promising progress on unimodal data imputation (e.g. image inpainting), models for multimodal data imputation are far from satisfactory. In this work, we propose variational selective autoencoder (VSAE) for this task. Learning only from partially-observed data, VSAE can model the joint distribution of observed/unobserved modalities and the imputation mask, ing in a unified model for various down-stream tasks including data generation and imputation. Evaluation on synthetic high-dimensional and challenging low-dimensional multimodal datasets shows significant improvement over state-of-the-art imputation models. Modern deep learning techniques rely heavily on extracting information from large scale datasets of clean and complete training data, such as labeled data or images with all pixels. Practically these data is costly due to the limited resources or privacy concerns. Having a model that learns and extracts information from partially-observed data will largely increase the application spectrum of deep learning models and provide benefit to down-stream tasks, e.g. data imputation, which has been an active research area. Despite promising progress, there are still challenges in learning effective imputation models: 1) Some prior works focus on learning from fully-observed data and then performing imputation on partially-observed data ; 2) They usually have strong assumptions on missingness mechanism (see Appendix A.1) such as data is missing completely at random (MCAR) ; 3) Some other works explore only unimodal imputation such as image in-painting for high-dimensional data . Modeling any combination of data modalities has not been well-established yet. This can limit the potential of such models since raw data in real-life is usually acquired in a multimodal manner . A class of prior works focus on learning the conditional likelihood of the modalities . However, they require complete data during training and cannot handle arbitrary conditioning. In practice, one or more of the modalities maybe be missing, leading to a challenging multimodal data imputation task. For more on related works, see Appendix A.2. The unimodal/multimodal proposal networks are employed by selection indicated by the arrows. Standard normal prior is ignored for simplicity. φ, ψ, θ and are the parameters of each modules. All components are trained jointly. We propose Variational Selective Autoencoder (VSAE) for multimodal data generation and imputation. It can model the joint distribution of data and mask and avoid the limited assumptions such as MCAR. VSAE is optimized efficiently with a single variational objective. The contributions are summarized as: A novel variational framework to learn from partially-observed multimodal data; VSAE can learn the joint distribution of observed/unobserved modalities and the mask, ing in a unified model for various down-stream tasks including data generation/imputation with relaxed assumptions on missigness mechanism; Evaluation on both synthetic high-dimensional and challenging low-dimensional multimodal datasets shows improvement over the state-of-the-art data imputation models. Problem Statement. Let x = [x 1, x 2 ..., x M] be the complete data with M modalities. The size of each x i may vary. A binary mask variable m ∈ {0, 1} M: m i = 1 indicates x i is observed and m i = 0 indicates x i is unobserved. The set of observed modalities O = {i|m i = 1} and unobserved modalities U = {i|m i = 0} are complementary. Accordingly, we denote the representation of observed and unobserved modalities with x o = [x i |m i = 1] and x u = [x i |m i = 0]. Assuming x and m are dependent, we aim to model the joint distribution p(x, m). As a , VSAE can be used for both imputation and generation. Proposed Model. The high-level overview of VSAE (see Figure 1) is that the multimodal data is encoded to a latent space factorized w.r.t. the modalities. The latent variable of each modality is selectively chosen between a unimodal encoder (if the modality is observed) or a multimodal encoder (if the modality is unobserved). Next all the modalities and mask are reconstructed by decoding the aggregated latent codes. Mathematically, we aim to model the joint distribution of the data x = [x o, x u] and mask m. Following VAE formulation (see Appendix A.3), we derive the ELBO for log p(x, m) with approximate posterior q(z|x, m): Decoder The probability distribution factorizes over modalities assuming that reconstructions are conditionally independent given complete latent variables of all modalities: Selective proposal distribution for encoders , we assume latent variables factorizes w.r.t the modalities. Given this, we define the proposal distribution for each modality as This is based on the intuitive assumption that the latent space of each modality is independent of the others given its data is observed. If the modality is missing, its latent variable is selectively inferred from other observed modalities. For partially-observed setting, x u is unavailable even during training. Thus, we define the objective function for training by taking expectation of the ELBO in Equation over x u. Only one term in Equation is dependent on x u, so the final objective is derived as We approximate E x j [log p θ (x j |m, z)], j ∈ U by sampling x j from the prior network (standard normal) and passing through the decoder. Our experiments show even a single sample is sufficient to learn the model effectively. In fact, the prior network can be used as a self-supervision mechanism to find the most likely samples which dominate the other samples when taking the expectation. See Appendix B for more details. We evaluate our model on high-dimensional multi-modal data and low-dimensional tabular data in comparison with state-of-the-art latent variable models. To test the robustness of our model, we evaluate our model under various challenging missingness mechanisms. Table 1: Data Imputation. Missing ratio is 0.5. For all lower is better. Last two rows are trained with fully-observed data. We show mean/std over 3 independent runs. ∆ ≤ 0.001. Low-dimensional tabular data. We choose UCI repository datasets (contains both numerical and categorical data, training/test split as 4/1 and 20% of training set for validation). We randomly sample from independent Bernoulli distributions with pre-defined missing ratio to generate masks that are fixed during training and test. In Table 1, we observe that VSAE trained with partially-observed data outperforms other baselines, even those models trained with fully-observed data on some datasets. We argue this is due to two potential reasons: the mask provides a natural way of dropout on the data space, thereby, helping the model to generalize; If the data is noisy or has outliers, which is common in low-dimensional data, learning from partially-observed data can improve the by ignoring these data. Figure 2 indicates VSAE are more robust to missing ratio. High-dimensional multimodal data. We synthesize two bimodal datasets using MNIST and SVHN datasets: Synthetic1 contains randomly paired two digits in MNIST as,,,,; Synthetic2 contains pairs of one digit in MNIST with a random same digit in SVHN. VSAE has better performance with lower variance (see Table 1). Experiments also indicate that VSAE is robust under different missing ratios, whereas other baselines are sensitive to the missing ratio, which is consistent as UCI experiments. We believe this is because of the underlying mechanism of proper proposal distribution selection and prior sharing. The separation of unimodal/multimodal encoders helps the model to attend to the observed data, while baselines only have single proposal distribution inferred from the whole input. Thus, VSAE can easily ignore unobserved noisy modalities and attends on observable useful modalities, but baselines rely on neural networks to extract useful information from the whole data (which is dominated by missing information in case of high missing ratio). Figure 3: Data Generation. Generated w/o conditional information. As shown, the correspondence between modalities (pre-defined pairs) are preserved. After training, we sample from prior to generate the data and mask. In UCI experiments, We calculate the proportion of 0 in generated mask vectors over 100 samples and average on all experiments, we get 0.3123 ± 0.026, 0.4964 ± 0.005, 0.6927 ± 0.013 for missing ratio of 0.3, 0.5, 0.7. It indicates VSAE can learn mask distribution. We observe that conditions on the reconstructed mask in the data decoders improve the performance. We believe this is because the mask vector can inform the data decoder about the missingness in data space since the latent space is shared by all modalities thereby allowing it to generate data from the selective proposal distribution. Figure 3 Following , the imputation process is to learn a generative distribution for unobserved missing data. To be consistent with notations in Section 2, let x = [x 1, x 2, ..., x M] be the complete data of all modalities where x i denote the feature representation for the i-th modality. We also define m ∈ {0, 1} M as the binary mask vector, where m i = 1 indicates if the i-th modality is observed, and m i = 0 indicates if it is unobserved: Given this, the observed data x o and unobserved data x u are represented accordingly: In the standard maximum likelihood setting, the unknown parameters are estimated by maximizing the following marginal likelihood, integrating over the unknown missing data values: characterizes the missingness mechanism p(m|x o, x u) in terms of independence relations between the complete data x = [x o, x u] and the mask m: • Missing completely at random (MCAR): • Missing at random (MAR): • Not missing at random (NMAR): Most previous data imputation methods works under MCAR or MAR assumptions since With such decoupling, we do not need missing information to marginalize the likelihood, and it provides a simple but approximate framework to learn from partially-observed data. Data Imputation. Classical imputation methods such as MICE and MissForest (Stekhoven and Bühlmann, 2011) learn discriminative models to impute missing features from observed ones. With recent advances in deep learning, several deep imputation models have been proposed based on autoencoders (? ;), generative adversarial nets (GANs) , and autoregressive models . GAN-based imputation method GAIN proposed by assumes that data is missing completely at random. Moreover, this method does not scale to high-dimensional multimodal data. Several VAE based data imputation methods (; ;) have been proposed in recent years. formulated variational autoencoders with arbitrary conditioning (VAEAC) for data imputation which allows generation of missing data conditioned on any combination of observed data. This algorithm needs complete data during training cannot learn from partially-observed data only. and modified VAE formulation to model the likelihood of the observed data only. However, they require training of a separate generative network for each dimension thereby increasing computational requirements. In contrast, our method aims to model joint distribution of observed and unobserved data along with the missingness pattern (imputation mask). This enables our model to perform both data generation and imputation even under relaxed assumptions on missingness mechanism (see Appendix A.1). A class of prior works such as conditional VAE and conditional multimodal VAE focus on learning the conditional likelihood of the modalities. However, these models requires complete data during training and cannot handle arbitrary conditioning. Alternatively, several generative models aim to model joint distribution of all modalities (; ?; ;). However, multimodal VAE based methods such as joint multimodal VAE and multimodal factorization model (MFM) require complete data during training. On the other hand, proposed another multimodal VAE (namely MVAE) can be trained with incomplete data. This model leverages a shared latent space for all modalities and obtains an approximate joint posterior for the shared space assuming each modalities to be factorized. However, if training data is complete, this model cannot learn the individual inference networks and consequently does not learn to handle missing data during test. Building over multimodal VAE approaches, our model aims to address the shortcomings above within a flexible framework. In particular, our model can learn multimodal representations from partially observed training data and perform data imputation from arbitrary subset of modalities during test. By employing a factorized multimodal representations in the latent space it resembles disentangled models which can train factors specialized for learning from different parts of data . Variational Autoencoder (VAE) is a probabilistic generative model, where data is constructed from a latent variable z with a prior distribution p(z). It is composed of an inference network and a generation network to encode and decode data. To model the likelihood of data, the true intractable posterior p(z|x) is approximated by a proposal distribution q φ (z|x), and the whole model is trained until ideally the decoded reconstructions from the latent codes sampled from the approximate posterior match the training data. In the generation module, p θ (x|z), a decoder realized by a deep neural network parameterized by θ, maps a latent variable z to the reconstructionx of observation x. In the inference module, an encoder parameterized by φ produces the sufficient statistics of the approximation posterior q φ (z|x) (a known density family where sampling can be readily done). In vanilla VAE setting, by simplifying approximate posterior as a parameterized diagonal normal distribution and prior as a standard diagonal normal distribution N (0, I), the training criterion is to maximize the following evidence lower bound (ELBO) w.r.t. θ and φ. where D KL denotes the Kullback-Leibler (KL) divergence. Usually the prior p(z) and the approximate q φ (z|x) are chosen to be in simple form, such as a Gaussian distribution with diagonal covariance, which allows for an analytic calculation of the KL divergence. While VAE approximates p(x), conditional VAE approximates the conditional distribution p(x|y). By simply introducing a conditional input, CVAE is trained to maximize the following ELBO: B.1. Architecture We construct each module of our model using neural networks and optimize the parameters via backpropagation techniques. Following the terms in standard VAE, our model is composed of encoders and decoders. The architecture is shown in Figure 1 with different modalities denoted by different colors. The data space of unobserved modalities is shaded to differentiate from observed modalities. The whole architecture can be viewed as an integration of two auto-encoding structures: the top-branch data-wise encoders/decoders and the bottom-branch mask-wise encoders/decoder. The selective proposal distribution chooses between the unimodal and multimodal encoders if the data is observed or not. The outputs of all encoders are aggregated and a common latent space is shared among all decoders. In the rest of this section we explain different modules in the proposed model. For more details about architecture and implementation see Appendix B. Selective Factorized Encoders Standard proposal distribution of VAEs depends on the whole data and can not handle incomplete input when the data is partially-observed. To overcome this, we introduce our selective proposal distribution, which is factorized with respect to the modalities. As defined in Equation, q φ (z i |x i), named as the unimodal proposal distribution, is inferred only from each observed individual modality of data. However, if the modality is unobserved, the multimodal proposal distribution q ψ (z i |x o, m) is used to infer corresponding latent variables from other observed modalities and mask. Hence, the learned model can impute the missing information by combining unimodal proposal distribution of observed modalities and multimodal proposal distribution of the unobserved modalities. The condition on the mask could make the model aware of the missing pattern and could help the model to attend to observed modalities. For each modality x i, we have a separate encoder to infer its unimodal proposal distribution parameterized by φ. For the multimodal proposal distribution, however, we use a single encoder parameterized by ψ. This encoder outputs the latent codes for all modalities, and we simply obtain the latent variable for each modality by slicing the output vector to M sequential vectors. We simply model all the proposal distributions as normal distributions by setting the outputs of all encoders as mean and variance of a normal distribution. For the unimodal proposal distributions, we have where µ φ and Σ φ are deterministic neural networks parameterized by φ that output the mean and covariance respectively. Similarly, the multimodal proposal distribution m) ) can be modeled by a neural network with x o and m as the inputs. The reparameterization in standard VAE is used for end-to-end training. Decoding through Latent Variable Aggregator F After selecting and sampling from proper proposal distributions for all modalities, the variational latent codes can be fed to the downstream decoders even when the observation is incomplete. To do this, the information from different modalities interact by aggregating their stochastic latent codes before going through the decoders: Here we simply choose the aggregator F(·) = concat(·), i.e., concatenating the latent codes as one single vector. One may also use other aggregation functions such as max/mean pooling or matrix fusion to combine latent codes from all modalities. The decoders take the shared aggregated variational latent codes as input to generate data and mask. Mask Vector Encoding and Decoding The mask variable m is encoded into the latent space through the multimodal proposal network. The latent space is shared by the mask and data decoders. The mask decoder is an MLP parameterized by in our implementation. It maps the aggregated latent codes from the selective proposal distributions to a reconstruction of the M -dimensional binary mask vector. We assume each dimension of the mask variable is an independent Bernoulli distribution. Training With reparameterization trick , we can jointly optimize the objective derived in Equation Equation with respect to these parameters defined above on training set: Since Equation only requires the mask and observed data during training, this modified ELBO L φ,θ,ψ, (x o, m) can be optimized without the presence of unobserved modalities. The KL-divergence term is calculated analytically for each factorized term. The conditional log-likelihood term is computed by negating reconstruction loss function. (See Section 3 and Appendix B.3.) Inference The learned model can be used for both data imputation and generation. For imputation, the observed modalities x o and mask m are fed through the encoders to infer the selective proposal distributions. Then the sampled latent codes are decoded to estimate the unobserved modalities x u. All the modules in Figure 1 are used for imputation. For generation, since no data is available at all, the latent codes are sampled from the prior and go through the decoders to generate the data and the mask. In this way, only modules after the aggregator are used without any inference modules. In all models, all the layers are modeled by MLP without any skip connections or resnet modules. Basically, the unimodal encoders take single modality data vector as input to infer the unimodal proposal distribution; the multimodal encoders take the observed data vectors and mask vector as as input to infer the multimodal proposal distributions. The input vector to multimodal encoders should have same length for the neural network. Here we just concatenate all modality vectors and replace the unobserved modality vectors with some noise. In UCI repository experiment, we replace the unobserved modality vectors as standard normal noise. In Bimodal experiment, we simply replace the pixels of unobserved modality as zero. Note that all the baselines has encoders/decoders with same or larger number of parameters than our method. We implement our model using PyTorch. Unimodal Encoders In UCI repository experiment, the unimodal encoders for numerical data are modeled by 3-layer 64-dim MLPs and the unimodal encoders for categorical data are modeled by 3-layer 64-dim MLPs, all followed by Batch Normalization and Leaky ReLU nonlinear activations. In MNIST+MNIST bimodal experiment, the unimodal encoders are modeled by 3-layer 128-dim MLPs followed by Leaky ReLU nonlinear activations; In MNIST+SVHN bimodal experiment, the unimodal encoders are modeled by 3-layer 512-dim MLPs followed by Leaky ReLU nonlinear activations. We set the latent dimension as 20-dim for every modality in UCI repository experiments and 256-dim for every modality in Bimodal experiments. UCI data unimodal encoder: Linear→ BatchNorm1d Multimodal Encoders In general, any model capable of multimodal fusion can be used here to map the observed data x o and the mask m to the latent variables z. However, in this paper we simply use an architecture similar to unimodal encoders. The difference is that the input to unimodal encoders are lower dimensional vectors of an individual modalities. But, the input to the multimodal encoders is the complete data vector with unobserved modalities replaced with noise or zeros. As the input to the multimodal encoders is the same for all modalities (i.e., q(z i |x o, m) ∀i), we model the multimodal encoders as one single encoder to take advantage of the parallel matrix calculation speed. Thus the multimodal encoder for every experiment has the same structure as its unmidal encoder but with full-dimensional input. Aggregator In our models, we simply use vector concatenation as the way of aggregating. We use Adam optimizer for all models. For UCI numerical experiment, learning rate is 1e-3 and use validation set to find a best model in 1000 epochs. For UCI categorical experiment, learning rate is 1e-2 and use validation set to find a best model in 1000 epochs. For bimodal experiments, learning rate is 1e-4 and use validation set to find a best model in 1000 epochs. All modules in our models are trained jointly. In our model, we calculate the conditional log-likelihood of unobserved modality by generating corresponding modalities from prior. We initially train the model for some (empirically we choose 20) epochs without calculating the conditional log-likelihood of x u. And then first feed the partially-observed data to the model and generate the unobserved modalitỹ x u without calculating any loss; then feed the same batch for another pass, calculate the conditional log-likelihood using real x o and generated x u as ground truth. Table 3: Imputation on Numerical datasets. Missing ratio is 0.5. Last two rows are trained with fully-observed data. Evaluated by NRMSE, lower is better. Table 4: Imputation on MNIST+MNIST. Missing ratio is 0.3, 0.5 and 0.7. Last two rows are trained with fully-observed data. Evaluated by combined errors of two modalities, lower is better. Table 6: Imputation on MNIST+SVHN. Missing ratio is 0.3, 0.5 and 0.7. Last two rows are trained with fully-observed data. Evaluated by combined errors of two modalities, lower is better.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We propose a novel VAE-based framework learning from partially-observed data for imputation and generation.

Conditional VAE on top of latent spaces of pre-trained generative models that enables  transfer between drastically different domains while preserving locality and semantic alignment.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: In many domains, especially enterprise text analysis, there is an abundance of data which can be used for the development of new AI-powered intelligent experiences to improve people's productivity. However, there are strong-guarantees of privacy which prevent broad sampling and labeling of personal text data to learn or evaluate models of interest. Fortunately, in some cases like enterprise email, manual annotation is possible on certain public datasets. The hope is that models trained on these public datasets would perform well on the target private datasets of interest. In this paper, we study the challenges of transferring information from one email dataset to another, for predicting user intent. In particular, we present approaches to characterizing the transfer gap in text corpora from both an intrinsic and extrinsic point-of-view, and evaluate several proposed methods in the literature for bridging this gap. We conclude with raising issues for further discussion in this arena. Using publicly available text data to train predictive models for use in privacy-aware enterprise settings is a very fruitful direction in the area of document understanding. However, when the labeled training dataset (source domain) is different from the unlabeled test (target domain), the two datasets likely follow different distributions. This application setting violates the i.i.d. assumption made by classic supervised learning methods and calls for domain adaptation techniques to properly account for this difference. State of the art domain adaptation techniques are generally developed and evaluated using a limited number of benchmark datasets and under constrained settings. The extent to which these methods are applicable for predictive settings over enterprise text data has neither been explored nor characterized in detail. To explore the effectiveness of state of the art domain adaptation methodology in enterprise text data, we focus on communication intent prediction in enterprise email. In particular, we use two public enterprise email datasets (Avocado, an IT company, and Enron, an oil company) to systematically analyze the transfer problem in enterprise email. The two intent prediction tasks that we focus on in this study are Meeting Intent (the email expresses an intent to meet with the recipient) and Commitment Intent (the email expresses an action the sender intends to take in the future) -both of which are binary classification tasks. Denote the joint distributions of the source and the target domain as P S (X, Y) and P T (X, Y) respectively. In our setting, we assume P S (X, Y) = P T (X, Y) and seek to establish measures that quantify these differences. This difference can be measured: directly (intrinsically) in terms of observed distributional differences of words, n-grams, or more generally the representation conditioned on class; or in terms of the downstream (extrinsic) impact on models learned to predict the class when used in a different domain. We refer to these measurable differences as the transfer gap. To understand the nature and extent to which these challenges persist in enterprise email, we first present intrinsic analyses of email text and features. We then proceed to extrinsic analyses to determine the extent to which state of the art methodology can account for the aforementioned challenges. Our contributions are: we provide ways of measuring the transfer gap intrinsically over both classical bag-of-words and bag-of-n-grams representations as well as distributed representations; provide evidence that these distributional differences lead to downstream measurable differences specifically in enterprise domains across different companies; and, provide a first evaluation of proposed transfer methods from elsewhere in the literature to enterprise settings. We first begin by analyzing the intrinsic differences in distribution (i.e. the transfer gap) across the two datasets. The way text is processed into features has evolved over the years from BoW vectors, to dense feature vectors from Skip-Gram word embeddings, and more recently feature vectors that take into account contextual information such as ELMO and BERT. We seek to establish distributional differences exist across a variety of representation choices. To this end, we first analyze the difference by first comparing most frequent words when conditioned on positive class (i.e. comparing the head of P (w|+)) and then comparing the contexts in which these most frequent words in the positive class are used. Then we proceed to examine the transfer gap in terms of distributional measures of sentence-level encodings using a generic CNN encoder and state of the art word embeddings. To analyze the most frequent words in positive intent, we compared the overlap between the top 30 n-grams (where n ranges from 1 to 3) in each dataset, when considering only sentences with positive labels, for each task. The for 1-grams show that the overlap for the most frequent words in the two enterprises is 53.3% and 70.0% for meeting and commitment intents respectively. Similar counts hold for bi-grams and tri-grams too. From this analysis, it is apparent that for both tasks there is a significant number of words (nearly half in Meeting and close to a third in Commitment) that are frequently used to express positive intent, which do not overlap across domains. Furthermore, the most frequent words in positive-intent sentences overlap more in Commitment than in Meeting Intent. This means that in Meeting Intent the difference in distribution may come down to different word usage to express intent, whereas in Commitment if differences exist they may be more subtle and due to differences in context. To explore whether the top-30 positive-intent associated words were used in different contexts, we embedded each sentence using contextual word embeddings. For each domain and task, we considered each word in the top-30 most frequent list for positive intent (obtained as described above), and we retrieved a distribution of its contextual word embeddings from each of the sentences in which the word was present. To measure whether there is a difference in terms of the contexts in which a word was used in one domain vs. another, we compared the distribution of its contextual word embeddings in the two domains by maximum mean discrepancy (MMD): Avocado (source) vs. Enron (target). MMD is a kernel-based symmetric measure of distribution difference that can be applied to a variety of representation types including embeddings; we use a Gaussian kernel. As a baseline reference since any sample even from the same distribution may show differences, we also performed this comparison with two disjoint subsets of the Avocado dataset. The are shown in Table 1 The previous intrinsic analyses consider lists of individual most frequent words. To capture sentencelevel structure, sentences in modern methodologies are represented as sequences of dense vectors, often obtained with a supervised deep encoder. In order to evaluate the extent to which there is a distribution difference in this dense sentence-level representation across domains, for each task and domain we trained a CNN encoder/classifier in that particular domain (for example Avocado). We then compared two distributions (in terms of MMD) of the CNN dense encodings of two sets of sentences from: the domain in which the encodings were trained; across domains. The can be found in Table 2. From these , one can appreciate that in all cases except one (Avocado-Enron for Commitment), the difference between distributions of two sets of sentences in the same vs. different domains is different by an order of magnitude. In the dense encoding representation, we see a large difference in the distribution across domains, and can expect a gap in the performance of predictive models when trained within vs. across domains. The main takeaways from the intrinsic analyses are: the transfer gap can be observed in individual words (n-grams) used in sentences that express intent, the transfer gap can be observed in terms of the context distribution in the top words that express intent, the transfer gap can be observed in terms of the sentence encodings of the sentences in which they occur. Therefore, an ideal transfer learning method to bridge the transfer gap would need to address these word-level and sentence-level differences. Now that we have established through the intrinsic analyses that there is a distributional difference, we analyze how that difference is reflected in terms of predictive performance of state of the art text classification and domain adaptation methods. We use two classes of methods: non-transfer text classification methods without accounting for the difference in distribution, and methods which perform domain adaptation. To provide a measure of the extrinsic transfer gap, we fix the training set and train each model on the training set and apply it in-domain and out-of-domain. This is equivalent to training on one enterprise's data and deploying the model both to that enterprise (in-domain) and to another enterprise (out-of-domain). Because the ratio of positive/negatives is different in the in-domain test set versus the out-of-domain test set, we use AUC as our performance metric which is a ranking-based metric that is invariant to class-skew. Thus it is reasonable to expect a similar performance on the out-of-domain test set on AUC if it really is from the same distribution as the in-domain. We prefer AUC over other performance metrics (like average precision and F1) since they are sensitive to class skew making it hard to use them for comparisons across test sets where the class skew changes. Transfer Methods The two domain adaptation methods that we use are: an mSDA autoencoder which combines unlabeled data from the source and target domains to learn a joint feature representation that can then be used with a regular classifier like logistic regression, and a domain adversarial deep learning method using a CNN architecture, which combines the labeled source domain data and the unlabeled target domain data, to extract a latent representation that is invariant across domains, and still useful for performing classification in the source domainwe call this CNN+ADV. Non-transfer baselines: as baselines, we apply models which are not designed to perform any transfer across domains, and inherently assume that the source and the target domains follows the same distribution. The non-transfer text classification methods we use are: L1-regularized logistic regression, which uses sparse BoW representation; A CNN, which encodes each sentence in a dense encoding using word embeddings as input -to provide a controlled comparison this is the same CNN as in CNN+ADV essentially eliminating the adversarial training; mSDA-NT where unlabeled data from only the source domain is used when training the autoencoder. Examining the in Table 3: AUC Scores for both Prediction Tasks. Each pair of adjacent rows in a block compares the model applied in-domain (upper row) and the model applied out-of-domain (lower row). The transfer out-of-domain (e.g. CNN+ADV, mSDA) can be compared to the corresponding no Transfer in-domain (e.g. CNN, mSDA-NT). Commitments. Referring back to the MMD on right of Table 2, note that MMD actually shows Enron is closer to Avocado than a sample of Avocado is! This can be interpreted as the variation in the language in commitments in Enron is smaller than that seen in Avocado; therefore a random sample of Enron commitments has less variability. Given that this predicts the direction of better out-of-domain performance, understanding how to leverage this better in learning is an interesting future direction. Now, note that both of the transfer methods generally improve over the logistic regression baseline but not over their most similar no-transfer counterpart. In much of the transfer learning literature a new model and training approach is introduced jointly and only compared to a baseline. Here we see by comparing to a no-transfer version, the gains on out-of-domain relative to a simple baseline is due to improved modeling but that it has not closed the size of the gap between in-domain and out-of-domain performance. In fact, many transfer papers do not compute in-domain performance nor compare on a performance metric that is skew invariant.

Represent the TLDR for finding the corresponding scientific article:

Insights on the domain adaptation challenge, when predicting user intent in enterprise email.

Represent this text to find a much much longer article about it with say hundreds-thousands of words.

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent the article for finding a single-sentence TLDR

Question: Hierarchical Reinforcement Learning is a promising approach to long-horizon decision-making problems with sparse rewards. Unfortunately, most methods still decouple the lower-level skill acquisition process and the training of a higher level that controls the skills in a new task. Treating the skills as fixed can lead to significant sub-optimality in the transfer setting. In this work, we propose a novel algorithm to discover a set of skills, and continuously adapt them along with the higher level even when training on a new task. Our main contributions are two-fold. First, we derive a new hierarchical policy gradient, as well as an unbiased latent-dependent baseline. We introduce Hierarchical Proximal Policy Optimization (HiPPO), an on-policy method to efficiently train all levels of the hierarchy simultaneously. Second, we propose a method of training time-abstractions that improves the robustness of the obtained skills to environment changes. Code and are available at sites.google.com/view/hippo-rl. Reinforcement learning (RL) has made great progress in a variety of domains, from playing games such as Pong and Go BID5 BID20 to automating robotic locomotion BID11 BID20 b), dexterous manipulation (b; BID1, and perception BID7 a). Yet, most work in RL is still learning a new behavior from scratch when faced with a new problem. This is particularly inefficient when dealing with tasks that are hard to solve due to sparse rewards or long horizons, or when solving many related tasks. A promising technique to overcome this limitation is Hierarchical Reinforcement Learning (HRL) BID17 a). In this paradigm, policies have several modules of abstraction, so the reuse of a subset of the modules becomes easier. The most common case consists of temporal abstraction BID9 ), where a higher-level policy (manager) takes actions at a lower frequency, and its actions condition the behavior of some lower level skills or sub-policies. When transferring knowledge to a new task, most prior works fix the skills and train a new manager on top. Despite having a clear benefit in kick-starting the learning in the new task, having fixed skills can considerably cap the final performance on the new task (a). Little work has been done on adapting pre-trained sub-policies to be optimal for a new task. In this paper, we develop a new framework for adapting all levels of temporal hierarchies simultaneously. First, we derive an efficient approximated hierarchical policy gradient. Our key insight is that, under mild assumptions, the manager's decisions can be considered part of the observation from the perspective of the sub-policies. This decouples the gradient with respect to the manager and the sub-policies parameters and provides theoretical justification for a technique used in other prior works . Second, we introduce an unbiased sub-policy specific baseline for our hierarchical policy gradient. Our experiments reveal faster convergence, suggesting efficient gradient variance reduction. Then we introduce a more stable way of using this gradient, Hierarchical Proximal Policy Optimization (HiPPO). This helps us take more conservative steps in our policy space BID12, necessary in hierarchies because of the interdependence of each layer. Finally we also evaluate the benefit of varying the time-commitment to the sub-policies, and show it helps both in terms of final performance and zero-shot adaptation to similar tasks. We define a discrete-time finite-horizon discounted Markov decision process (MDP) by a tuple M = (S, A, P, r, ρ 0, γ, H), where S is a state set, A is an action set, P: S × A × S → R + is the transition probability distribution, γ ∈ is a discount factor, and H Figure 1. Temporal hierarchy studied in this paper. A latent code zt is sampled from the manager policy π θ h (zt|st) every p time-steps, using the current observation s kp. The actions at are sampled from the sub-policy π θ l (at|st, z kp) conditioned on the same latent code from timestep t = kp to timestep (k + 1)p − 1 the horizon. Our objective is to find a stochastic policy π θ that maximizes the expected discounted reward within the MDP, η(DISPLAYFORM0 We denote by τ = (s 0, a 0, ...,) the entire state-action trajectory, where s 0 ∼ ρ 0 (s 0), a t ∼ π θ (a t |s t), and s t+1 ∼ P(s t+1 |s t, a t). Prior works have been focused on learning a manager that combines provided sub-policies, but they do not further train the sub-policies when learning a new task. However, preventing the skills from learning in sub-optimal behavior in new tasks. This effect is exacerbated when the skills were learned in a task agnostic way or in a different environment. In this paper, we present a HRL method that learns all levels of abstraction in the hierarchical policy: the manager learns to make use of the low-level skills, while the skills are continuously adapted to attain maximum performance in the given task. We derive a policy gradient update for hierarchical policies that monotonically improves the performance. Furthermore, we demonstrate that our approach prevents sub-policy collapse behavior, when the manager ends up using just one skill, observed in previous approaches. When using a hierarchical policy, the intermediate decision taken by the higher level is not directly applied in the environment. This consideration makes it unclear how it should be incorporated into the Markovian framework of RL: should it be treated as an observed variable, like an action, or as a latent?In this section, we first prove that one framework is an approximation of the other under mild assumptions. Then, we derive an unbiased baseline for the HRL setup that reduces its variance. Thirdly, we introduce the notion of information bottleneck and trajectory compression, which proves critical for learning reusable skills. Finally, with these findings, we present our method, Hierarchical Proximal Policy Optimization (HiPPO), an on-policy algorithm for hierarchical policies that monotonically improves the RL objective, allowing learning at all levels of the policy and preventing sub-policy collapse. Policy gradient algorithms are based on the likelihood ratio trick BID21 to estimate the gradient of returns with respect to the policy parameters as DISPLAYFORM0 In the context of HRL, a hierarchical policy with a manager π θ h (z t |s t) selects every p time-steps one of n sub-policies to execute. These sub-policies, indexed by z ∈ [n], can be represented as a single conditional probability distribution over actions π θ l (a t |z t, s t). This allows us to also leverage skills learned with Stochastic Neural Networks (SNNs) (a). Under this framework, the probability of a trajectory τ = (s 0, a 0, s 1, . . ., s H) can be written as DISPLAYFORM1 The mixture action distribution, which presents itself as an additional summation over skills, prevents the additive factorization when taking the logarithm, as in Eq. 1. This can yield considerable numerical instabilities due to the product of the p sub-policy probabilities. For instance, in the case where all the skills are distinguishable all the sub-policies probabilities but one will have small values, ing in an exponentially small value. In the following Lemma, we derive an approximation of the policy gradient, whose error tends to zero as the skills become more diverse, and draw insights on the interplay of the manager actions. Lemma 1. If the skills are sufficiently differentiated, then the latent variable can be treated as part of the observation to compute the gradient of the trajectory probability. Let π θ h (z|s) and π θ l (a|s, z) be Lipschitz functions w.r.t. their parameters, and assume that 0 < π θ l (a|s, z j) < ∀j = kp, DISPLAYFORM2 Proof. See Appendix. Our assumption is that the skills are diverse. Namely, for each action there is just one sub-policy that gives it high probability. In this case, the latent variable can be treated as part of the observation to compute the gradient of the trajectory probability. Many algorithms to extract lowerlevel skills are based on promoting diversity among the skills (a;), so our assumption usually holds. We further empirically analyze this assumption in the appendix. The REINFORCE policy gradient estimate is known to have large variance. A very common approach to mitigate this issue without biasing the estimate is to subtract a baseline from the returns BID8. We show how, under the assumptions of Lemma 1, we can formulate an unbiased latent dependent baseline for the approximate gradient (Eq. 4).Lemma 2. For any functions b h: S → R and b l: S ×Z → R we have: DISPLAYFORM0 Proof. See Appendix. Now we apply Lemma 1 and Lemma 2 to Eq. 1. By using the corresponding value functions as the function baseline, the return can be replaced by the Advantage function BID11, and we obtain the following gradient expression: DISPLAYFORM1 This hierarchical policy gradient estimate has lower variance than without baselines, but using it for policy optimization through stochastic gradient descent still yields an unstable algorithm. In the next section, we further improve the stability and sample efficiency of the policy optimization by incorporating techniques from Proximal Policy Optimization BID12. Using an appropriate step size in policy space is critical for stable policy learning. We adopt the approach used by Proximal Policy Optimization (PPO) BID12, which modifies the cost function in a way that prevents large changes to the policy while only requiring the computation of the likelihood. Letting r h,kp (θ) = DISPLAYFORM0, and using the super-index clip to denote the clipped objective version, we obtain the new surrogate objective: DISPLAYFORM1 We call this algorithm Hierarchical Proximal Policy Optimization (HiPPO). Next, we introduce two critical additions: a switching of the time-commitment between skills, and an information bottleneck at the lower-level. Both are detailed in the following subsections. Most hierarchical methods either consider a fixed timecommitment to the lower level skills (a;), or implement the complex options framework BID9 BID2. In this work we propose an in-between, where the time-commitment to the skills is a random variable sampled from a fixed distribution Categorical(T min, T max) just before the manager takes a decision. This modification does not hinder final performance, and we show it improves zero-shot adaptation to a new task. This approach to sampling rollouts is detailed given in Algorithm 1 in the appendix. If we apply the above HiPPO algorithm in the general case, there is little incentive to either learn or maintain a diverse set of skills. We claim this can be addressed via two simple additions:• Let z only take a finite number of values • Provide a masked observation to the skills DISPLAYFORM0 The masking function f restricts the information about the task, such that a single skill cannot perform the full task. We use a hard-coded agent-space and problem-space split (; a) that hides all task-related information and only allows the sub-policies to see proprioceptive information. With this setup, all the missing information needed to perform the task must come from the sequence of latent codes passed to the skills. We can interpret this as a lossy compression, whereby the manager encodes the relevant problem information into log n bits sufficient for the next p timesteps. We design the experiments to answer the following questions: 1) How does HiPPO compare against a flat policy when learning from scratch? 2) Does it lead to more robust policies? 3) How well does it adapt already learned skills? and 4) Does our skill diversity assumption hold in practice? Figure 4. Benefit of adapting some given skills when the preferences of the environment are different from those of the environment where the skills were originally trained. In this section, we study the benefit of using the HiPPO algorithm instead of standard PPO on a flat policy BID12. The , shown in FIG0, demonstrate that training from scratch with HiPPO leads faster learning and better performance than flat PPO. Furthermore, the benefit of HiPPO does not just come from having temporally correlated exploration, as PPO with action repeat converges at a performance level well below our method. Finally, FIG1 shows the effectiveness of using the presented baseline. For this task, we take 6 pre-trained subpolicies encoded by a Stochastic Neural Network BID18 that were trained in a diversity-promoting environment (a). We fine-tune them with HiPPO on the Gather environment, but with an extra penalty on the velocity of the Center of Mass. This can be understood as a preference for cautious behavior. This requires adjustment of the sub-policies, which were trained with a proxy reward encouraging them to move as far as possible (and hence quickly). Fig. 4 shows the difference between fixing the sub-policies and only training a manager with PPO vs using HiPPO to simultaneously train a manager and fine-tune the skills. The two initially learn at the same rate, but HiPPO's ability to adjust to the new dynamics allows it to reach a higher final performance. In this paper, we examined how to effectively adapt hierarchical policies. We began by deriving a hierarchical policy gradient and approximation of it. We then proposed a new method, HiPPO, that can stably train multiple layers of a hierarchy. The adaptation experiments suggested that we can optimize pretrained skills for downstream environments, and learn emergent skills without any unsupervised pre-training. We also explored hierarchy from an information bottleneck point of view, demonstrating that HiPPO with randomized period can learn from scratch on sparsereward and long time horizon tasks, while outperforming non-hierarchical methods on zero-shot transfer. There are many enticing avenues of future work. For instance, replacing the manually designed bottleneck with a variational autoencoder with an information bottleneck could further improve HiPPO's performance and extend the gains seen here to other tasks. Also, as HiPPO provides a policy architecture and gradient expression, we could explore using meta-learning on top of it in order to learn better skills that are more useful on a distribution of different tasks. The key points in HRL are how the different levels of the hierarchy are defined, trained, and then re-used. In this work, we are interested in approaches that allow us to build temporal abstractions by having a higher level taking decisions at a slower frequency than a lower-level. There has been growing interest in HRL for the past few decades BID17 BID9, but only recently has it been applied to high-dimensional continuous domains as we do in this work .To obtain the lower level policies, or skills, most methods exploit some additional assumptions, like access to demonstrations (; BID4 BID10 BID13, policy sketches BID0, or task decomposition into sub-tasks (; BID16 . Other methods use a different reward for the lower level, often constraining it to be a "goal reacher" policy, where the signal from the higher level is the goal to reach BID6 BID3 BID20 . These methods are very promising for state-reaching tasks, but might require access to goal-reaching reward systems not defined in the original MDP, and are more limited when training on tasks beyond state-reaching. Our method does not require any additional supervision, and the obtained skills are not constrained to be goal-reaching. When transferring skills to a new environment, most HRL methods keep them fixed and simply train a new higher-level on top . Other work allows for building on previous skills by constantly supplementing the set of skills with new ones BID14, but they require a hand-defined curriculum of tasks, and the previous skills are never fine-tuned. Our algorithm allows for seamless adaptation of the skills, showing no trade-off between leveraging the power of the hierarchy and the final performance in a new task. Other methods use invertible functions as skills , and therefore a fixed skill can be fully over-written when a new layer of hierarchy is added on top. This kind of "fine-tuning" is promising, although they do not apply it to temporally extended skills as we are interested in here. One of the most general frameworks to define temporally extended hierarchies is the options framework BID17, and it has recently been applied to continuous state spaces BID2. One of the most delicate parts of this formulation is the termination policy, and it requires several regularizers to avoid skill collapse BID2 BID19. This modification of the objective may be difficult to tune and affects the final performance. Instead of adding such penalties, we propose having skills of a random length, not controlled by the agent during training of the skills. The benefit is two-fold: no termination policy to train, and more stable skills that transfer better. Furthermore, these works only used discrete action MDPs. We lift this assumption, and show good performance of our algorithm in complex locomotion tasks. The closest work to ours in terms of final algorithm is the one proposed by. Their method can be included in our framework, and hence benefits from our new theoretical insights. We also introduce two modifications that are shown to be highly beneficial: the random time-commitment explained above, and the notion of an information bottleneck to obtain skills that generalize better. Algorithm 1 Collect Rollout 1: Input: skills π θ l (a|s, z), manager π θ h (z|s), time-commitment bounds P min and P max, horizon H, and bottleneck function o = f (s) 2: Reset environment: s 0 ∼ ρ 0, t = 0. 3: while t < H do Sample time-commitment p ∼ Cat([P min, P max]) Sample skill z t ∼ π θ h (·|s t) 6: DISPLAYFORM0 Observe new state s t +1 and reward r t 9: end for 10:t ← t + p 11: end while 12: Output: DISPLAYFORM1 Input: skills π θ l (a|s, z), manager π θ h (z|s), horizon H, learning rate α while not done do for actor = 1, 2,..., N do Obtain trajectory with Collect Rollout Estimate advantagesÂ(a t, o t, z t) and To answer the posed questions, we evaluate our new algorithms on a variety of robotic navigation tasks. Each task is a different robot trying to solve the Gather environment , depicted in FIG2, in which the agent must collect apples (green balls, +1 reward) while avoiding bombs (red balls, -1 reward). This is a challenging hierarchical task with sparse rewards that requires agents to simultaneously learn perception, locomotion, and higher-level planning capabilities. We use 2 different types of robots within this environment. Snake is a 5-link robot with a 17-dimensional observation space and 4-dimensional action space; and Ant a quadrupedal robot with a 27-dimensional observation space and 8-dimensional action space. Both can move and rotate in all directions, and Ant faces the added challenge of avoiding falling over irrecoverably. DISPLAYFORM0 Gather Algorithm Initial Mass Dampening Inertia Friction Snake Flat PPO 2.72 3.16 (+16%) 2.75 (+1%) 2.11 (-22%) 2.75 (+1%) HiPPO, p = 10 4.38 3.28 (-25%) 3.27 (-25%) 3.03 (-31%) 3.27 (-25%) HiPPO random p 5.11 4.09 (-20%) 4.03 (-21%) 3.21 (-37%) 4.03 (-21%) Ant Flat PPO 2.25 2.53 (+12%) 2.13 (-5%) 2.36 (+5%) 1.96 (-13%) HiPPO, p = 10 3.84 3.31 (-14%) 3.37 (-12%) 2.88 (-25%) 3.07 (-20%) HiPPO random p 3.22 3.37 (+5%) 2.57 (-20%) 3.36 (+4%) 2.84 (-12%) Table 1. Zero-shot transfer performance of flat PPO, HiPPO, and HiPPO with randomized period. The performance in the initial environment is shown, as well as the average performance over 25 rollouts in each new modified environment. We try several different modifications to the base Snake Gather and Ant Gather environments. One at a time, we change the body mass, dampening of the joints, body inertia, and friction characteristics of both robots. The , presented in Table 1, show that HiPPO with randomized period Categorical([T min, T max]) not only learns faster initially on the original task, but it is also able to better handle these dynamics changes. In terms of the percent change in policy performance between the training environment and test environment, it is able to outperform HiPPO with fixed period on 6 out of 8 related tasks without even taking any gradient steps. Our hypothesis is that the randomized period teaches the policy to adapt to wide variety of scenarios, while its information bottleneck is able to keep separate its representations for planning and locomotion, so changes in dynamics aren't able to simultaneously affect both. Gather Algorithm Cosine Similarity max z =z kp π θ l (a t |o t, z) Table 2. Empirical evaluation of Lemma 1. On the right column we evaluate the quality of our assumption by computing what is the average largest probability of a certain action under other skills. On the left column we report cosine similarity between our approximate gradient and the gradient computed using Eq. 2 without approximation. In Lemma 1, we assumed that the sub-policies present ought to be diverse. This allowed us to derive a more efficient and numerically stable gradient. In this section, we empirically test the validity of our assumption, as well as the quality of our approximation. For this we run, on Snake Gather and Ant Gather, the HiPPO algorithm both from scratch and on some pretrained skills as described in the previous section. In Table 2, we report the average maximum probability under other sub-policies, corresponding to from the assumption. We observe that in all settings this is on the order of magnitude of 0.1. Therefore, under the p = 10 that we use in our experiments, the term we neglect has a factor p−1 = 10 −10. It is not surprising then that the average cosine similarity between the full gradient and the approximated one is almost 1, as also reported in Table 2. We only ran two random seeds of these experiments, as the seemed pretty consistent, and they are more computationally challenging to run. For all experiments, both PPO and HiPPO used learning rate 3 × 10 −3, clipping parameter = 0.1, 10 gradient updates per iteration, a batch size of 100,000, and discount γ = 0.999. HiPPO used n = 6 sub-policies. Ant Gather has a horizon of 5000, while Snake Gather has a horizon of 8000 due to its larger size. All runs used three random seeds. HiPPO uses a manager network with 2 hidden layers of 32 units, and a skill network with 2 hidden layers of 64 units. In order to have roughly the same number of parameters for each algorithm, flat PPO uses a network with 2 hidden layers with 256 and 64 units respectively. For HiPPO with randomized period, we resample p ∼ Uniform{5, 15} every time the manager network outputs a latent, and provide the number of timesteps until the next latent selection as an input into both the manager and skill networks. The single baselines and skill-dependent baselines used a MLP with 2 hidden layers of 32 units to fit the value function. The skill-dependent baseline receives, in addition to the full observation, the active latent code and the time remaining until the next skill sampling. Lemma 1. If the skills are sufficiently differentiated, then the latent variable can be treated as part of the observation to compute the gradient of the trajectory probability. Concretely, if π θ h (z|s) and π θ l (a|s, z) are Lipschitz in their parameters, and 0 < π θ l (a t |s t, z j) < ∀j = kp, then DISPLAYFORM0 Proof. From the point of view of the MDP, a trajectory is a sequence τ = (s 0, a 0, s 1, a 1, . . ., a H−1, s H). Let's assume we use the hierarchical policy introduced above, with a higher-level policy modeled as a parameterized discrete distribution with n possible outcomes π θ h (z|s) = Categorical θ h (n). We can expand P (τ) into the product of policy and environment dynamics terms, with z j denoting the jth possible value out of the n choices, DISPLAYFORM1 Taking the gradient of log P (τ) with respect to the policy parameters θ = [θ h, θ l], the dynamics terms disappear, leaving: DISPLAYFORM2 The sum over possible values of z prevents the logarithm from splitting the product over the p-step sub-trajectories. This term is problematic, as this product quickly approaches 0 as p increases, and suffers from considerable numerical instabilities. Instead, we want to approximate this sum of products by a single one of the terms, which can then be decomposed into a sum of logs. For this we study each of the terms in the sum: the gradient of a sub-trajectory probability under a specific latent ∇ θ π θ h (z j |s kp) (k+1)p−1 t=kp π θ l (a t |s t, z j). Now we can use the assumption that the skills are easy to distinguish, 0 < π θ l (a t |s t, z j) < ∀j = kp. Therefore, the probability of the sub-trajectory under a latent different than the one that was originally sampled z j = z kp, is upper bounded by p. Taking the gradient, applying the product rule, and the Lipschitz continuity of the policies, we obtain that for all z j = z kp, Thus, we can across the board replace the summation over latents by the single term corresponding to the latent that was sampled at that time. DISPLAYFORM3 ∇ θ log P (τ) = Interestingly, this is exactly ∇ θ P (s 0, z 0, a 0, s 1, . . .). In other words, it's the gradient of the probability of that trajectory, where the trajectory now includes the variables z as if they were observed. Lemma 2. For any functions b h: S → R and b l: S × Z → R we have: Then, we can write out the definition of the expectation and undo the gradient-log trick to prove that the baseline is unbiased. ∇ θ log π s,θ (a t |s t, z kp)b(s t, z kp)] = 0 DISPLAYFORM4 We'll follow the same strategy to prove the second equality: apply the same law of iterated expectations trick, express the expectation as an integral, and undo the gradient-log trick.

We propose HiPPO, a stable Hierarchical Reinforcement Learning algorithm that can train several levels of the hierarchy simultaneously, giving good performance both in skill discovery and adaptation.

represent this text to find a much much longer article about it with say hundreds-thousands of words

A backward model of previous (state, action) given the next state, i.e. P(s_t, a_t | s_{t+1}), can be used to simulate additional trajectories terminating at states of interest! Improves RL learning efficiency.