Question: Represent the scientific article to find a short TLDR

Question: Employing deep neural networks as natural image priors to solve inverse problems either requires large amounts of data to sufficiently train expressive generative models or can succeed with no data via untrained neural networks. However, very few works have considered how to interpolate between these no- to high-data regimes. In particular, how can one use the availability of a small amount of data (even 5-25 examples) to one's advantage in solving these inverse problems and can a system's performance increase as the amount of data increases as well? In this work, we consider solving linear inverse problems when given a small number of examples of images that are drawn from the same distribution as the image of interest. Comparing to untrained neural networks that use no data, we show how one can pre-train a neural network with a few given examples to improve reconstruction in compressed sensing and semantic image recovery problems such as colorization. Our approach leads to improved reconstruction as the amount of available data increases and is on par with fully trained generative models, while requiring less than 1% of the data needed to train a generative model. We study the problem of recovering an image x x x 0 ∈ R n from m linear measurements of the form y y y 0 = A A Ax x x 0 + η η η ∈ R m where A A A ∈ R m×n is a known measurement operator and η η η ∈ R m denotes the noise in our system. Problems of this form are ubiquitous in various domains ranging from image processing, machine learning, and computer vision. Typically, the problem's difficulty is a of its ill-posedness due to the underdetermined nature of the system. To resolve this ambiguity, many approaches enforce that the image must obey a natural image model. While traditional approaches typically use hand-crafted priors such as sparsity in the wavelet basis, recent approaches inspired by deep learning to create such natural image model surrogates have shown to outperform these methods. Deep Generative Priors: Advancements in generative modelling have allowed for deep neural networks to create highly realistic samples from a number of complex natural image classes. Popular generative models to use as natural image priors are latent variable models such as Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs). This is in large part due to the fact that they provide a low-dimensional parameterization of the natural image manifold that can be directly exploited in inverse imaging tasks. When enforced as a natural image prior, these models have shown to outperform traditional methods and provide theoretical guarantees in problems such as compressed sensing, phase retrieval, and blind deconvolution/demodulation. However, there are two main drawbacks of using deep generative models as natural image priors. The first is that they require a large amount of data to train, e.g., hundreds of thousands of images to generate novel celebrity faces. Additionally, they suffer from a non-trivial representation error due to the fact that they model the natural image manifold through a low-dimensional parameterization. Untrained Neural Network Priors: On the opposite end of the data spectrum, recent works have shown that randomly initialized neural networks can act as natural image priors without any learning. first showed this to be the case by solving tasks such as denoising, inpainting, and super-resolution via optimizing over the parameters of a convolutional neural network to fit to a single image. The showed that the neural network exhibited a bias towards natural images, but due to the high overparameterization in the network, required early stopping to succeed. A simpler model was later introduced in which was, in fact, underparameterized and was able to both compress images while solving various linear inverse problems. Both methods require no training data and do not suffer from the same representation error as generative models do. Similar to generative models, they have shown to be successful image priors in a variety of inverse problems. Based on these two approaches, we would like to investigate how can one interpolate between these data regimes in a way that improves upon work with untrained neural network priors and ultimately reaches or exceeds the success of generative priors. More specifically, we would like to develop an algorithm that 1) performs just as well as untrained neural networks with no data and 2) improves performance as the amount of provided data increases. We introduce a framework to solve inverse problems given a few examples (e.g., 5 − 25) drawn from the same data distribution as the image of interest (e.g., if the true image is of a human face, the examples are also human faces). Our main contributions are the following: • We show how one can pre-train a neural network using a few examples drawn from the data distribution of the image of interest. Inspired by, we propose to jointly learn a latent space and parameters of the network to fit to the examples that are given and compare the use of an 2 reconstruction loss and a Maximum Mean Discrepancy (MMD) loss. • We then propose to solve the inverse problem via a two-step process. We first optimize over the pre-trained network's latent space. Once a solution is found, we then refine our estimate by optimizing over the latent space and parameters of the network jointly to improve our solution. found this method to work well in the case when the network is a fully trained generative model, and we show here that even a pre-trained neural network from a small number of examples can benefit from such an approach. • We show that our approach improves upon untrained neural networks in compressed sensing even with as few as 5 examples from the data distribution and exhibits improvements as the number of examples increases. We also show that semantics can be learned from these few examples in problems such as colorization where untrained neural networks fail. With only 100 examples, our model's performance is competitive with fully trained generative models. Related work: We mention that there has been previous work in investigating how to use a small amount of data to help solve the compressed sensing problem. The authors use an untrained neural network as a natural image prior and, when given a small amount of data, adopt a learned regularization term when solving the inverse problem. This term is derived by posing the recovery problem as a Maximum a Posteriori (MAP) estimation problem and by placing a Gaussian prior on the weights of the untrained network. While we have not compared our method to this learned regularization approach here, we aim to do so in a subsequent manuscript. We consider the problem of recovering an image x x x 0 ∈ R n from noisy linear measurements of the form y y y 0 = A A Ax x x 0 + η η η ∈ R m where A A A ∈ R m×n and m n. We also assume that x x x 0 is drawn from a particular data distribution D and that we are given a low number of examples drawn from the same distribution, i.e., x x x 0 ∼ D and we are given x x x i ∼ D where i ∈ [S]. Here and throughout this work, we refer to these examples drawn from D as low shots. We propose using the range of a deep neural network as a natural image model. In particular, we model the image x x x 0 as the output of a neural network G(z z z; θ θ θ), where z z z ∈ R k is a latent code and θ θ θ ∈ R P are the parameters of the network. Pre-training: Prior to solving the inverse problem, we propose to first pre-train the network using the low shots that are given. More specifically, we fit the weights and input of the neural network to the low shots to provide a crude approximation to the data distribution underlying the image of interest. Given low shots {x, we aim to find latent codes {z z z i} S i=1 and parameters θ θ θ that solve where L: R n × R n → R is a loss function. We investigate the use of different loss functions in a later section. The ing optimal parameters found are denoted byθ θ θ,ẑ z z 1,...,ẑ z z S. Solving the inverse problem: Using the weights found via pre-training, we begin solving the inverse problem by first optimizing over the latent code space to find an approximate solution: We investigated different ways to initialize the latent code and found that sampling from a multivariate Gaussian distribution fit using {ẑ z z i} S i=1 was sufficient. Note here that we keep the parameters of the network fixed after training. The intuition is that we want to use the semantics regarding the data distribution learned via pre-training the network's parameters and find the optimal latent code that corresponds to the image of interest. Once the optimal latent codeẑ z z is found, we then refine our solution by solving withθ θ θ andẑ z z as our initial iterates. The ing parameters θ θ θ 0 and z z z 0 give our final estimate: Losses to learn the data distribution: We discuss two loss functions that we considered in our experiments to learn semantics regarding the underlying data distribution. The first is a simple 2 reconstruction loss to promote data fidelity, i.e., we pre-train our network by solving While used a combination of the Laplacian-L1 loss and 2 loss, we found the 2 loss to work well. The second loss is an estimate of the kernel MMD for comparing two probability distributions using only finitely many samples. In our case, given a kernel k(·, ·) 2 and low shots x x x j ∼ D for j ∈ [S], we want to find parameters and S inputs that solve the following: We compare the success of these two loss functions in the following section. We now consider solving inverse problems with our approach and compare to three different baselines: an untrained neural network, optimizing the latent space of a trained Wasserstein GAN with gradient penalty, and the image-adaptivity approach of (IAGAN). Each method uses the same DCGAN architecture with a latent code dimension of 128. In each problem, the image of interest is from a hold-out test set from the CelebA dataset. The GAN was trained on a corpus of over 200, 000 64 × 64 celebrity images and our low-shot models were trained on small (5 − 100 images) subsets of this. Compressed Sensing: We first consider the compressed sensing problem where we want to recover an image x x x 0 ∈ R n from random Gaussian measurements of the form y y y 0 = A A Ax x x 0 ∈ R m where A A A ∈ R m×n has i.i.d. N entries with m n. We refer to amount of undersampling m n as the compression ratio. We trained our models using the two different loss functions proposed in the previous section for various numbers of shots S ∈. We note that as the number of shots increases, our method continues to improve and we see comparable performance between our method with only 100 shots and optimizing over the latent code space of a fully trained GAN. While the 2 trained nets perform slightly better than the MMD trained nets for low numbers of shots, the MMD trained nets improve more steadily and consistently as the number of shots increases. While we expect IAGAN to be superior due to being trained with over 200, 000 images, the MMD trained model's performance with 100 images is not far behind. We note that for higher numbers of measurements, the untrained neural network's performance surpasses that of our 5 shot models. This is mainly due to the fact that we optimized the untrained neural network's parameters for a longer period of time and with a higher learning rate than our low shot models in this easier setting. We now consider an inverse problem that requires an understanding of the underlying data's semantics: the colorization task. Here we want to recover an RGB image x x x 0 ∈ R 64×64×3 from its grayscale version y y y 0 = A A Ax x x 0 ∈ R 64×64. The operator A A A mixes the color channels of the image via the ITU-R 601-2 luma transform (the same transform used by the Python Imaging Library (PIL)). Untrained neural networks clearly fail in solving this type of problem since they have no prior information regarding the data distribution. We compare our model trained with 10 shots using the MMD loss to the various baselines in Figure 2. Note that even with only 10 previous examples, our model does not fall prey to the usual issues with using untrained neural networks for colorization. Our algorithm provides faithful image reconstructions that are even on par with a trained GAN. This document was prepared as an account of work sponsored by an agency of the United States government. Neither the United States government nor Lawrence Livermore National Security, LLC, nor any of their employees makes any warranty, expressed or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately owned rights. Reference herein to any specific commercial product, process, or service by trade name, trademark, manufacturer, or otherwise does not necessarily constitute or imply its endorsement, recommendation, or favoring by the United States government or Lawrence Livermore National Security, LLC. The views and opinions of authors expressed herein do not necessarily state or reflect those of the United States government or Lawrence Livermore National Security, LLC, and shall not be used for advertising or product endorsement purposes.

Represent the TLDR for finding the corresponding scientific article

We show how pre-training an untrained neural network with as few as 5-25 examples can improve reconstruction results in compressed sensing and semantic recovery problems like colorization.

Represent the short summary for finding the corresponding scientific article (long).

We introduce a type of neural network that is structurally resistant to adversarial attacks, even when trained on unaugmented training sets.  The resistance is due to the stability of network units wrt input perturbations.


Question: Represent the scientific article to find a short TLDR

Question: We propose Cooperative Training (CoT) for training generative models that measure a tractable density for discrete data. CoT coordinately trains a generator G and an auxiliary predictive mediator M. The training target of M is to estimate a mixture density of the learned distribution G and the target distribution P, and that of G is to minimize the Jensen-Shannon divergence estimated through M. CoT achieves independent success without the necessity of pre-training via Maximum Likelihood Estimation or involving high-variance algorithms like REINFORCE. This low-variance algorithm is theoretically proved to be superior for both sample generation and likelihood prediction. We also theoretically and empirically show the superiority of CoT over most previous algorithms in terms of generative quality and diversity, predictive generalization ability and computational cost. Generative modeling is essential in many scenarios, including continuous data modeling (e.g. image generation BID6, stylization BID17, semisupervised classification BID13) and sequential discrete data modeling (e.g. neural text generation BID2).For discrete data with tractable density like natural language, generative models are predominantly optimized through Maximum Likelihood Estimation (MLE), inevitably introducing exposure bias BID14, which in that given a finite set of observations, the optimal parameters of the model trained via MLE do not correspond to the ones maximizing the generative quality. Specifically, the model is trained on the data distribution of inputs and tested on a different distribution of inputs, namely, the learned distribution. This discrepancy implies that in the training stage, the model is never exposed to its own errors and thus in the test stage, the errors made along the way will quickly accumulate. On the other hand, for general generative modeling tasks, an effective framework, named Generative Adversarial Network (GAN) BID6, was proposed to train an implicit density model for continuous data. GAN introduces a discriminator D φ parametrized by φ to distinguish the generated samples from the real ones. As is proved in BID6, GAN essentially optimizes an approximately estimated Jensen-Shannon divergence (JSD) between the currently learned distribution and the target distribution. GAN shows promising in many unsupervised and semi-supervised learning tasks. The success of GAN in the naissance of a new paradigm of deep generative models, i.e. adversarial networks. However, since the gradient computation requires backpropagation through the generator's output, GAN can only model the distribution of continuous variables, making it non-applicable for generating discrete sequences like natural language. Researchers then proposed Sequence Generative Adversarial Network (SeqGAN), which uses model-free policy gradient algorithm to optimize the original GAN objective. With SeqGAN, the expected JSD between current and target discrete data distribution is minimized if the training is perfect. SeqGAN shows observable improvements in many tasks. Since then, many variants of SeqGAN have been proposed to improve its performance. Nonetheless, SeqGAN is not an ideal algorithm for this problem, and current algorithms based on it cannot show stable, reliable and observable improvements that covers all scenarios, according to a previous survey. The detailed reason will be discussed in detail in Section 2.In this paper, we propose Cooperative Training (CoT), a novel, low-variance, bias-free algorithm for training likelihood-based generative models on discrete data by directly optimizing a wellestimated Jensen-Shannon divergence. CoT coordinately trains a generative module G, and an auxiliary predictive module M, called mediator, for guiding G in a cooperative fashion. For theoretical soundness, we derive the proposed algorithm directly from the definition of JSD. We further empirically and theoretically demonstrate the superiority of our algorithm over many strong baselines in terms of generative performance, generalization ability and computational performance in both synthetic and real-world scenarios. Notations. P denotes the target data distribution. θ denotes the parameters of the generative module G. φ denotes the parameters of the auxiliary predictive mediator module M. Any symbol with subscript g and m stands for that of the generator and mediator, respectively. s stands for a complete sample from the training dataset or a generated complete sequence, depending on the specific context. s t means the t-length prefix of the original sequence, i.e. an incomplete sequence of length t. x denotes a token, and x t stands for a token that appears in the t-th place of a sequence. Thus s t = [x 0, x 1, x 2, . . ., x t−1] while the initial case s 0 is ∅. Maximum likelihood estimation is equivalent to minimizing the KL divergence using the samples from the real distribution:min DISPLAYFORM0 where G θ (s) is the estimated probability of s by G θ and p data is the underlying real distribution. Limitations of MLE. MLE is essentially equivalent to optimizing a directed Kullback-Leibler (KL) divergence between the target distribution P and the currently learned distribution G, denoted as KL(P G). However, since KL divergence is asymmetric, given finite observations this target is actually not ideal. As stated in BID0, MLE tries to minimize DISPLAYFORM1 • When P (s) > 0 and G(s) → 0, the KL divergence grows to infinity, which means MLE assigns an extremely high cost to the "mode dropping" scenarios, where the generator fails to cover some parts of the data.• When G(s) > 0 and P (s) → 0, the KL divergence shrinks to 0, which means MLE assigns an extremely low cost to the scenarios, where the model generates some samples that do not locate on the data distribution. Likewise, optimizing KL(G P) will lead to exactly the reversed problems of the two situations. An ideal solution is to optimize a symmetrized and smoothed version of KL divergence, i.e. the Jensen-Shannon divergence (JSD), which is defined as DISPLAYFORM2 where M = 1 2 (P + G). However, directly optimizing JSD is conventionally considered as an intractable problem. JSD cannot be directly evaluated and optimized since the equally interpolated distribution M is usually considered to be unconstructable, as we only have access to the learned model G instead of P. SeqGAN incorporates two modules, i.e. the generator and discriminator, parametrized by θ and φ respectively, as in the settings of GAN. By alternatively training these two modules, SeqGAN optimizes such an adversarial target: min Collect two equal-sized mini-batch of samples {sg} and {sp} from G θ and P, respectively 5: DISPLAYFORM0 Mix {sg} and {sp} as {s} 6:Update mediator M φ with {s} via Eq. 7: end for 8:Generate a mini-batch of sequences {s} ∼ G θ 9:Update generator G θ with {s} via Eq. 10: until CoT convergesThe objectives of generator G θ and discriminator D φ in SeqGAN can be formulated as DISPLAYFORM1 Discriminator: DISPLAYFORM2 where s ∼ G θ = [x 1, ..., x n] denotes a complete sequence sampled from the generator and the action value BID16, the fact that SeqGAN is essentially based on model-free reinforcement learning makes it a non-trivial problem for SeqGAN to converge well. As a , SeqGAN usually gets stuck in some fake local optimals. Specifically, although the discriminator can distinguish the samples from the generator easily, it is not able to effectively guide the generator because of the vanishing gradient, as is discussed in a recent survey. Although this problem can be alleviated by reshaping the reward signals based on the relative rankings of the outputs in a mini-batch BID11 BID8, they are more technical workarounds than essential solutions. DISPLAYFORM3 Second, SeqGAN trained via REINFORCE suffers from the "mode collapse" problem, which is similar to the original GAN. That is to say, the learned distribution "collapses" to the other side of KL divergence, i.e. KL(G P), which leads to the loss of diversity of generated samples. In other words, SeqGAN trains the model for better generative quality at the cost of diversity.3 COOPERATIVE TRAINING To be consistent with the goal that the target distribution should be well-estimated in both quality and diversity senses, an ideal algorithm for such models should be able to optimize a symmetric divergence or distance. For sequential discrete data modeling, since the data distribution is decomposed into a sequential product of finite-dimension multinomial distributions (always based on the softmax form), the failures of effectively optimizing JSD when the generated and real data distributions are distant, as discussed in, will not appear. As such, to optimize JSD is feasible. However, to our knowledge, no previous algorithms provide a direct, low-variance optimization of JSD. In this paper, we propose Cooperative Training (CoT), as shown in Algorithm 1, to directly optimize a well-estimated unbiased JSD for training such models. Each iteration of Cooperative Training mainly consists of two parts. The first part is to train a mediator M φ, which is a density function that estimates a mixture distribution of the learned generative distribution G θ and target latent distribution P = p data as DISPLAYFORM0 Since the mediator is only used as a density prediction module during training, the directed KL divergence is now free from so-called exposure bias for optimization of M φ. Denote DISPLAYFORM1 Lemma 1 (Mixture Density Decomposition) DISPLAYFORM2 By Lemma 1, for each step, we can simply mix balanced samples from training data and the generator, then train the mediator via Maximum Likelihood Estimation with the mixed samples. The objective J m (φ) for the mediator M parametrized by φ therefore becomes DISPLAYFORM3 Since the objective of MLE is bias-free for predictive purposes, the estimated M φ is also bias-free when adopted for estimating JSD. The training techniques and details will be discussed in Section 4.After each iteration, the mediator is exploited to optimize an estimated Jensen-Shannon divergence for G θ: DISPLAYFORM4 Note that the gradient Eq. should be performed for only one step because once G θ is updated the current mediator's estimation M φ becomes inaccurate. For any sequence or prefix of length t, we have: DISPLAYFORM5 DISPLAYFORM6 The detailed derivations can be found in the supplementary material. Note that Lemma 2 can be applied recursively. That is to say, given any sequence s t of arbitrary length t, optimizing s t's contribution to the expected JSD can be decomposed into optimizing the first term of Eq. FORMULA0 and solving an isomorphic problem for s t−1, which is the longest proper prefix of s t. When t = 1, since in Markov decision process the probability for initial state s 0 is always 1.0, it is trivial to prove that the final second term becomes 0.Therefore, Eq. can be reduced through recursively applying Lemma 2. After removing the constant multipliers and denoting the predicted probability distribution over the action space, i.e. G θ (·|s t) and M φ (·|s t), as π g (s t) and π m (s t) respectively, the gradient ∇ θ J g (θ) for training generator via Cooperative Training can be formulated as DISPLAYFORM7 For tractable density models with finite discrete action space in each step, the practical effectiveness of this gradient is well guaranteed for the following reasons. First, with a random initialization of the model, the supports of distributions G θ and P are hardly disjoint. Second, the first term of Eq. FORMULA0 is to minimize the cross entropy between G and M *, which tries to enlarge the overlap of two distributions. Third, since the second term of Eq. FORMULA0 is equivalent to maximizing the entropy of G, it encourages the support of G to cover the whole action space, which avoids the case of disjoint supports between G and P.The overall objective of CoT can be formulated as finding the maximal entropy solution of max DISPLAYFORM8 Note the strong connections and differences between the optimization objective of CoT FORMULA0 and that of GAN. FIG0 illustrates the whole Cooperative Training process. CoT has theoretical guarantee on its convergence. Theorem 3 (Jensen-Shannon Consistency) If in each step, the mediator M φ of CoT is trained to be optimal, i.e. M φ = M * = 1 2 (G θ + P), then optimization via Eq. leads to minimization of JSD(G P).Proof. Let p denote the intermediate states. It would be used in the detailed proof. All we need to show is DISPLAYFORM0 By inversely applying Lemma 2, the left part in Eq. can be recovered as DISPLAYFORM1 which is equivalent to DISPLAYFORM2 Since now mediator is trained to be optimal, i.e. M φ = M *, we have DISPLAYFORM3 This means training through CoT leads to minimization ofĴSD(P G θ). When the mediator is trained to be optimal,ĴSD(P G θ) = JSD(P G θ). This verifies the theorem. CoT has several practical advantages over previous methods, including MLE, Scheduled Sampling (SS) BID3 and adversarial methods like SeqGAN.First, although CoT and GAN both aim to optimize an estimated JSD, CoT is exceedingly more stable than GAN. This is because the two modules, namely generator and mediator, have similar tasks, i.e. to approach the same data distribution generatively and predictively. The superiority of CoT over inconsistent methods like Scheduled Sampling is obvious, since CoT theoretically guarantees the training effectiveness. Compared with methods that require pre-training in order to reduce variance like SeqGAN, CoT is computationally cheaper. More specifically, under recommended settings, CoT has the same order of computational complexity as MLE.Besides, CoT works independently. In practice, it does not require model pre-training via conventional methods like MLE. This is the first time that unbiased unsupervised learning is achieved on sequential discrete data without using supervised approximation for variance reduction or sophisticated smoothing as in Wasserstein GAN with gradient penalty (WGAN-GP) BID7. An interesting problem is to ask why we need to train a mediator by mixing the samples from both sources G and P, instead of directly training a predictive modelP on the training set via MLE. There are basically two points to interpret this. To apply the efficient training objective 13, one needs to obtain not only the mixture density model M = 1 2 (P + G) but also its decomposed form in each timestep i.e. M φ (s) = n t=1 M φ (s t |s t−1), without which the term π m (s t) in Eq 13 cannot be computed efficiently. This indicates that if we directly estimate P and compute M = 1 2 (G + P), the obtained M will be actually useless since its decomposed form is not available. Besides, as a derivative problem of "exposure bias", there is no guarantee for the modelP to work well on the generated samples i.e. s ∼ G θ to guide the generator towards the target distribution. Given finite observations, the learned distributionP is trained to provide correct predictions for samples from the target distribution P. There is no guarantee thatP can stably provide correct predictions for guiding the generator. Ablation study is provided in the appendix. Following the synthetic data experiment setting in, we design a synthetic Turing test, in which the negative log-likelihood NLL oracle from an oracle LSTM is calculated for evaluating the quality of samples from the generator. Particularly, to support our claim that our method causes little mode collapse, we calculated NLL test, which is to sample an extra batch of samples from the oracle, and to calculate the negative log-likelihood measured by the generator. We show that under this more reasonable setting, our proposed algorithm reaches the state-of-the-art performance with exactly the same network architecture. Note that models like LeakGAN BID8 contain architecture-level modification, which is orthogonal to our approach, thus will not be included in this part. The are shown in TAB2. Computational Efficiency Although in terms of time cost per epoch, CoT does not achieve the state-of-the-art, we do observe that CoT is remarkably faster than previous RL-GAN approaches. Besides, consider the fact that CoT is a sample-based optimization algorithm, which involves time BID3 8.89 8.71/-(MLE) (The same as MLE) 32.54 ± 1.14s Professor Forcing BID10 9 To show the hyperparameter robustness of CoT, we compared it with the similar as were evaluated in SeqGAN. DISPLAYFORM0 cost in sampling from the generator, this is acceptable. The also verifies our claim that CoT has the same order (i.e. the time cost only differs in a constant multiplier or extra lower order term) of computational complexity as MLE.Hyper-parameter Robustness. We perform a hyper-parameter robustness experiment on synthetic data experiment. When compared with the of similar experiments as in SeqGAN, our approach shows less sensitivity to hyper-parameter choices, as shown in FIG1. Note that since in all our attempts, the evaluated JSD of SeqGAN fails to converge, we evaluated NLL oracle for it as a replacement. Self-estimated Training Progress Indicator. Like the critic loss, i.e. estimated Earth Mover Distance, in WGANs, we find that the training loss of the mediator, namely balanced NLL, can be a real-time training progress indicator as shown in FIG2. Specifically, in a wide range, balanced NLL is a good estimation of real JSD(G P) with a steady translation, namely, balanced N LL = JSD(G P) + H(G) + H(P). 2.900 (σ = 0.025) 3.118 (σ = 0.018) 3.122 RankGAN BID11 As an important sequential data modeling task, zero-prior text generation, especially long and diversified text generation, is a good testbed for evaluating the performance of a generative model. Following the experiment proposed in LeakGAN BID8, we choose EMNLP 2017 WMT News Section as our dataset, with maximal sentence length limited to 51. We pay major attention to both quality and diversity. To keep the comparison fair, we present two implementations of CoT, namely CoT-basic and CoT-strong. As for CoT-basic, the generator follows the settings of that in MLE, SeqGAN, RankGAN and MaliGAN. As for CoT-strong, the generator is implemented with the similar architecture in LeakGAN.For quality evaluation, we evaluated BLEU on a small batch of test data separated from the original dataset. For diversity evaluation, we evaluated the estimated Word Mover Distance BID9, which is calculated through training a discriminative model between generated samples and real samples with 1-Lipschitz constriant via gradient penalty as in WGAN-GP BID7. To keep it fair, for all evaluated models, the architecture and other training settings of the discriminative models are kept the same. The are shown in TAB4 and TAB5. In terms of generative quality, CoT-basic achieves state-of-the-art performance over all the baselines with the same architecture-level capacity, especially the long-term robustness at n-gram level. CoT-strong using a conservative generation strategy, i.e. setting the inverse temperature parameter α higher than 1, as in BID8 achieves the best performance over all compared models. In terms of generative diversity, the show that our model achieves the state-of-the-art performance on all metrics including NLL test, which is the optimization target of MLE. We proposed Cooperative Training, a novel training algorithm for generative modeling of discrete data. CoT optimizes Jensen-Shannon Divergence, which does not have the exposure bias problem as the forward KLD. Models trained via CoT shows promising in sequential discrete data modeling tasks, including sample quality and the generalization ability in likelihood prediction tasks. B SAMPLE COMPARISON AND DISCUSSION TAB6 shows samples from some of the most powerful baseline models and our model. • CoT produces remarkably more diverse and meaningful samples when compared to Leak-GAN.• The consistency of CoT is significantly improved when compared to MLE. The Optimal Balance for Cooperative Training We find that the same learning rate and iteration numbers for the generator and mediator seems to be the most competitive choice. As for the architecture choice, we find that the mediator needs to be slightly stronger than the generator. For the best in the synthetic experiment, we adopt exactly the same generator as other compared models and a mediator whose hidden state size is twice larger (with 64 hidden units) than the generator. Theoretically speaking, we can and we should sample more batches from G θ and P respectively for training the mediator in each iteration. However, if no regularizations are used when training the mediator, it can easily over-fit, leading the generator's quick convergence in terms of KL(G θ P) or NLL oracle, but divergence in terms of JSD(G θ P). Empirically, this could be alleviated by applying dropout techniques BID15 with 50% keeping ratio before the output layer of RNN. After applying dropout, the empirical show good consistency with our theory that, more training batches for the mediator in each iteration is always helpful. However, applying regularizations is not an ultimate solution and we look forward to further theoretical investigation on better solutions for this problem in the future. " I think it was alone because I can do that, when you're a lot of reasons, " he said. It's the only thing we do, we spent 26 and $35(see how you do is we lose it," said both sides in the summer. CoT We focus the plans to put aside either now, and which doesn't mean it is to earn the impact to the government rejected. The argument would be very doing work on the 2014 campaign to pursue the firm and immigration officials, the new review that's taken up for parking. This method is true to available we make up drink with that all they were willing to pay down smoking. The number of people who are on the streaming boat would study if the children had a bottle -but meant to be much easier, having serious ties to the outside of the nation. However, they have to wait to get the plant in federal fees and the housing market's most valuable in tourism. MLE after the possible cost of military regulatory scientists, chancellor angela merkel's business share together a conflict of major operators and interest as they said it is unknown for those probably 100 percent as a missile for britain. but which have yet to involve the right climb that took in melbourne somewhere else with the rams even a second running mate and kansas. " la la la la 30 who appeared that themselves is in the room when they were shot her until the end " that jose mourinho could risen from the individual. when aaron you has died, it is thought if you took your room at the prison fines of radical controls by everybody, if it's a digital plan at an future of the next time. Possible Derivatives of CoT The form of equation 13 can be modified to optimize other objectives. One example is the backward KLD (a.k.a. Reverse KLD) i.e. KL(G P). In this case, the objective of the so-called "Mediator" and "Generator" thus becomes:"Mediator", now it becomes a direct estimatorP φ of the target distribution P: DISPLAYFORM0 Generator: DISPLAYFORM1 Such a model suffers from so-called mode-collapse problem, as is analyzed in Ian's GAN Tutorial BID5. Besides, as the distribution estimatorP φ inevitably introduces unpredictable behaviors when given unseen samples i.e. samples from the generator, the algorithm sometimes fails (numerical error) or diverges. In our successful attempts, the algorithm produces similar (not significantly better than) as CoT. The quantitive are shown as follows: Although under evaluation of weak metrics like BLEU, if successfully trained, the model trained via Reverse KL seems to be better than that trained via CoT, the disadvantage of Reverse KL under evaluation of more strict metric like eWMD indicates that Reverse KL does fail in learning some aspects of the data patterns e.g. completely covering the data mode.

We proposed Cooperative Training, a novel training algorithm for generative modeling of discrete data.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We show that posterior collapse in linear VAEs is caused entirely by marginal log-likelihood (not ELBO). Experiments on deep VAEs suggest a similar phenomenon is at play.


Question: represent the article for finding a single-sentence tldr

Question: Intrinsic rewards in reinforcement learning provide a powerful algorithmic capability for agents to learn how to interact with their environment in a task-generic way. However, increased incentives for motivation can come at the cost of increased fragility to stochasticity. We introduce a method for computing an intrinsic reward for curiosity using metrics derived from sampling a latent variable model used to estimate dynamics. Ultimately, an estimate of the conditional probability of observed states is used as our intrinsic reward for curiosity. In our experiments, a video game agent uses our model to autonomously learn how to play Atari games using our curiosity reward in combination with extrinsic rewards from the game to achieve improved performance on games with sparse extrinsic rewards. When stochasticity is introduced in the environment, our method still demonstrates improved performance over the baseline. Methods encouraging agents to explore their environment by rewarding actions that yield unexpected are commonly referred to as curiosity (Schmidhuber (1991; 1990a; b) ). Using curiosity as an exploration policy in reinforcement learning has many benefits. In scenarios in which extrinsic rewards are sparse, combining extrinsic and intrinsic curiosity rewards gives a framework for agents to discover how to gain extrinsic rewards. In addition, when agents explore, they can build more robust policies for their environment even if extrinsic rewards are readily available . These policies learned through exploration can give an agent a more general understanding of the of their actions so that the agent will have a greater ability to adapt using their existing policy if their environment changes. Despite these benefits, novelty-driven exploration methods can be distracted by randomness. (b;) When stochastic elements are introduced in the environment, agents may try to overfit to noise instead of learning a deterministic model of the effect of their own actions on their world. In particular, Burda et al. (2018a) showed that when a TV with white noise is added to an environment in which an agent is using the intrinsic curiosity module (ICM) developed by , the agent stops exploring the environment and just moves back and forth in front of the TV. In this paper, we present a new method for agent curiosity which provides robust performance in sparse reward environments and under stochasticity. We use a conditional variational autoencoder to develop a model of our environment. We choose to develop a conditional variational autoencoder (CVAE) due to the success of this architecture in modeling dynamics shown in the video prediction literature . We incorporate additional modeling techniques to regularize for stochastic dynamics in our perception model. We compute our intrinsic reward for curiosity by sampling from the latent space of the CVAE and computing an associated conditional probability which is a more robust metric than the commonly used pixel-level reconstruction error. The primary contributions of our work are the following. 1. Perception-driven approach to curiosity. We develop a perception model which integrates model characteristics proven to work well for deep reinforcement learning with recent architectures for estimating dynamics from pixels. This combination retains robust-ness guarantees from existing deep reinforcement learning models while improving the ability to capture complex visual dynamics. 2. Bayesian metric for surprise. We use the entropy of the current state given the last state as a measurement for computing surprise. This Bayesian approach will down-weight stochastic elements of the environment when learning a model of dynamics. As a , this formulation is robust to noise. For our experiments, autonomous agents use our model to learn how to play Atari games. We measure the effectiveness of our surprise metric as a meaningful intrinsic reward by tracking the total achieved extrinsic reward by agents using a combination of our intrinsic reward with extrinsic rewards to learn. We show that the policy learned by a reinforcement learning algorithm using our surprise metric outperforms the policies learned by alternate reward schemes. Furthermore, we introduce stochasticity into the realization of actions in the environment, and we show that our method still demonstrates successful performance beyond that of the baseline method. Perception-Driven Curiosity. Several existing models incentivize curious agent behavior through estimating and seeking visual novelty. and generalize count-based exploration traditionally used in tabular settings for continuous states. Burda et al. (2018b) learns a predictive model on features given by a randomly initialized target network and uses reconstruction error of the random features as intrinsic reward. provides a full review of recent perception-driven methods to encourage curiosity. In this work, we combine a CVAE, an architecture recently used to successfully estimate dynamics from image frames, with methodological approaches from deep reinforcement learning to build a robust perception model in which visual novelty is computed via an estimation of the conditional log-likelihood of observed states. Prediction-Based Exploration Bonuses. proposed an approach to exploration by building prediction models and formulating intrinsic reward as the error of the next state prediction. Recently, this line of work has been shown to explore efficiently in a large number of simulated environments by and Burda et al. (2018a). formalizes the prediction error as Bayesian surprise given a heteroscedastic Gaussian predictive model. This approach is closest to our own. However, in contrast to these reward methods which are built upon simple predictive models, our formulation of Bayesian surprise is computed via importance sampling from our latent variable model. This construction of surprise is significant due to the ability of this variational inference approach to express complex multimodal distributions over images. Information-Theoretic Measures for Exploration. Several methods rely on maximizing information-theoretic measures of agent behavior. The method by maximizes the entropy of agent trajectories between successive states directly. propose to learn skills without supervision by maximizing the mutual information between a latent skill embedding and the behaviour that the associated skill produces. introduce a method for unsupervised goal-conditional reinforcement learning which maximizes the entropy of the distribution of possible goals. uses the KL divergence between the previous and current dynamics models as intrinsic reward as a proxy for information gain. We indirectly measure the entropy of agent trajectories by measuring surprise in terms of the entropy of next state given the previous state, providing an alternative to these existing approaches. Video Prediction with Latent Variable Models. introduced a stochastic model for sequential data based on variational inference approaches by and. This model was adopted for high-dimensional data such as video by;; and. A similar model based on was used for next frame prediction in. We leverage the success of variational inference techniques for high-dimensional data to construct a stochastic model of videos from which surprise can be efficiently estimated. We construct a perception model for our agents using a conditional variational autoencoder (CVAE) which generates an estimation of an embedded state, φ t+1, given the embedded state itself, φ t+1. This generative model is conditioned on the last embedded state, φ t, and action, a t. Intuitively, this construction gives an agent a visual model of the environment conditioned on the dynamics associated with their interactions with the environment. The state embeddings are encoded by a neural network submodule in our architecture which computes feature vectors φ t and φ t+1 from states s t and s t+1 respectively. In our experiments, the states we observe from our simulation environment are image frames. We derive our approach and the following properties of our model from the theoretical properties of conditional variational autoencoders presented by. We first define p(φ t+1 |z, φ t, a t) as the generative distribution from which we draw the output φ t+1. The prior distribution of the latent space is given by p(z|φ t, a t) which is relaxed in the CVAE formulation to make the latent variable, z, statistically independent of the input variables. Thus, our prior for the latent space distribution is given by p(z) ∼ N (0, I). Through training, our model learns a latent representation. The distribution of this representation, q(z|φ t+1, φ t, a t), approximates p(z|φ t, a t). Using the previously defined distributions and the analysis by , we define the empirical lower bound of the conditional log-likelihood and objective function, f, of our CVAE as We recall that the sum of log-probabilities is equal to the reconstruction loss of our model up to a constant and multiplicative offset. Thus, we denote We can write the KL-divergence term in Equation 1 as The final component of our perception model is a neural network predicting a t from φ t and φ t+1 built off the inverse model presented by. This component regularizes for dynamics in the environment which do not contribute to agent actions. We note that this component controls for environment stochasticity when learning the weights of our network. The error in action prediction can be formulated in terms of maximum likelihood estimation of the parameters of our network under a multinomial distribution. This error is used as the loss for our action-prediction network and denoted L A. We can now use the approximation of our CVAE objective function and the loss from our actionprediction network to formulate the total loss for our perception model as We tune the hyper-parameters λ 1, λ 2, and λ 3 to weight the contribution of each loss in our model. The tuning procedure and hyperparameters used in our experiments are given in Appendix B. The full architecture of our perception model is shown Figure 1.b. We define Bayesian surprise as the amount an agent should be curious about an observation derived from a conditional likelihood of the observation occurring given the current world model of the agent. From our definition of curiosity, we want to reward actions more strongly which in less likely outcomes. Therefore, we use the negative of this conditional probability estimate as a reward for agents. In our approach, this probabilistic reward takes the form r t = − log p(φ t+1 |φ t, a t). Similar objectives were used in prior work which considered simple homoscedastic (Burda et al. (2018a) ) or heteroscedastic Gaussian forward models. Due to our use of a base CVAE architecture, our perception model can capture multimodal distributions over images. To retain this improved expressiveness in our derived intrinsic reward, we use importance sampling from the latent space of our CVAE to estimate conditional likelihoods for our formulation of surprise as follows. = log E (z∼q(z|φt+1,φt,at)) p(φ t+1 |z, φ t, a t)p(z) q(z|φ t+1, φ t, a t) ≥ E (z∼q(z|φt+1,φt,at)) log p(φ t+1 |z, φ t, a t)p(z) q(z|φ t+1, φ t, a t) We use the reconstruction loss of our model to compute the conditional probability log p(φ t+1 |z, φ t, a t). We recall that the negative logarithm of our conditional probability is equal to Bayesian surprise, so we explicitly define our reward as follows. We use the Bayesian surprise computed by our perception model as intrinsic reward input to a reinforcement learning algorithm. The interaction of this reward and our perception model with the reinforcement learning procedure is visualized in Figure 1.a. We evaluate the ability of our model to enable effective and robust exploration. We use Atari video games as simulation environments since they provide reasonably complex visual environments with large variations in both sparsity of extrinsic reward and stochasticity in scenes between different games. As a , Atari games have been frequently used as a testbed for curiosity approaches. (; a; b;) We use our intrinsic reward measurement with the proximal policy optimization (PPO) reinforcement learning algorithm developed by due to the ability of PPO to perform well with relatively little hyperparameter tuning. In training, we combine our intrinsic reward with extrinsic rewards provided by the game environments for task-specific success such as knocking blocks out of a wall in Breakout. We compare the ability of agents using this reward combination to learn to play different Atari games against the ability of agents using a leading alternate prediction-based exploration bonus by in combination with extrinsic rewards. We also compare our approach to agent behavior derived from policies learned by purely extrinsic rewards. Note that, though combination rewards are used to train PPO, each method is evaluated by comparing extrinsic reward per episode alone since extrinsic rewards measure the successful accomplishment of tasks in each game. The hyperparameters used in training as well as additional implementation details are given in Appendix B. Furthermore, Appendix A shows details and analysis of the perception model performance throughout training via this active learning procedure. We first test the impact of our curiosity-reward on learning to play Atari games with varying levels of extrinsic reward sparseness. Gravitar, Beam Rider, Breakout, Space Invaders, and River Raid all have reasonably dense extrinsic rewards. In contrast, Montezuma's Revenge, Pitfall, and Private Eye all have sparse extrinsic rewards and are thus known as a traditionally challenging games for deep reinforcement learning algorithms to play successfully with only the information provided by game scene observations. Our for training 3 seeds for each method over 10 million timesteps in each of these games are plotted in Figure 2. Table 1 summarizes the of the extrinsic rewards achieved at the end of training. The best performance for each game is bold in the respective row. For games with dense extrinsic rewards, the best performance is split somewhat equally between each of the 3 reward strategies. Thus, we conclude that we perform comparably to ICM in the case where extrinsic rewards are readily available. However, we also conclude that the value of using curiosity for the purpose of achieving improved performance in these cases is substantially less significant than when rewards are sparse considering that we also show comparable performance to using extrinsic reward only in these games. We recall the games with sparse extrinsic rewards are Pitfall, Montezuma's Revenge, and Private Eye. Though PPO achieved the highest extrinsic reward on Pitfall, no successful game strategy was found. The mean extrinsic reward for each method is always negative in Pitfall. Exploration necessary to ultimately discover a successful strategy may require incurring temporary negative extrinsic rewards, so comparing the average before any working strategy is learned by any method is premature. On both Montezuma's Revenge and Private Eye, our method outperforms the combination of ICM and extrinsic rewards as well as extrinsic rewards only. We further observe that the variance of our method at convergence in Private Eye is very low. These demonstrate the improved ability of our approach to use information from better models for perceiving complex scene dynamics in order to learn in the absence of extrinsic rewards. There is a challenging balance with methods rewarding novelty to encourage exploration of the unknown while avoiding confounding the model by focusing on randomness. We showed that our method explores more effectively than our baseline in cases where extrinsic rewards are sparse. However, we now need to demonstrate that that improved performance did not introduce brittleness to environmental noise. Following Burda et al. (2018b), we perform a sticky action experiment to demonstrate the robustness of our model to stochasticity. With a 25% probability, the action an agent takes is repeated over 8 consecutive frames in the environment though the agent believes that their new action decisions are being executing across those frames. We observe that the performance of ICM and our model in the presence of sticky actions in comparable with slightly better performance from our model. Our for training 3 seeds for each method over 10 million timesteps in each of these games are plotted in Figure 2. Table 2 summarizes the of the extrinsic rewards achieved at the end of training. The best performance for each game is bold in the respective row. In summary, we presented a novel method to compute curiosity through the use of a meaningfully constructed model for perception. We used a conditional variational autoencoder (CVAE) to learn scene dynamics from image and action sequences and computed an intrinsic reward for curiosity via a conditional probability derived from importance sampling from the latent space of our CVAE. In our experiments, we demonstrated that our approach allows agents to learn to accomplish tasks more effectively in environments with sparse extrinsic rewards without compromising robustness to stochasticity. We show robustness to stochasticity in our action space which we support through the actionprediction network used in our perception model. However, robustness to stochasticity in scenes is a separate challenge which the method we use as our baseline, ICM, cannot handle well. (a) Stochasticity in scenes occurs when there are significant changes between sequential image frames which are random with respect to agent actions. We hypothesize that this stochasticity requires a different approach to handle. A consideration in comparing models for curiosity and exploration in deep reinforcement learning is that typically both the dynamics model and intrinsic reward metric are constructed and compared as unit as we did in this paper. However, a conditional probability estimation could be derived the dynamics model given by ICM just as reconstruction error could be used as intrinsic reward from our CVAE. Alternately, other metrics measuring novelty and learning such as the KL divergence between sequential latent distributions in our model have been proposed in a general manner by. An interesting direction for future work would be to explore the impact of intrinsic reward metrics for curiosity on robustness to stochasticity in scenes independent across different choices of dynamics model. We analyze the ability of an agent using our model to perceive the environment. We confirm that our perception model generates image embeddings which can successfully construct realistic image frames from our video games. We also show that increasing ability to perceive the environment is associated with decreasing intrinsic rewards. To set up this experiment, we built a visual decoder to reconstruct images from the embeddings learned by the visual encoder shown in Figure 1.b. We trained our decoder on the game images (s t) and the image embeddings (φ t). Then, we used our decoder to reconstruct images from our predicted image embeddings (φ t). We chose to execute this experiment on Kung Fu Master since it is one of the more visually complex Atari games. Figure 4: Relationship between reconstruction error and intrinsic reward. Figure 5 shows the reconstructed images from our perception model next to the image frames which are our states. We note that the observations provided by the OpenAI Gym Atari game simulations are grayscale and rescaled to size 84x84. These images are what we input into our visual encoder. The reconstructions of our predicted embeddings are shown next to these images along with measurements of reconstruction error between the embedded images and intrinsic reward associated with that time step. We observe that the high intrinsic rewards are associated with poor reconstructions of the image frames. Analogously, low intrinsic rewards are associated with good reconstructions of the image frames. We note that all of the scenes presented in this visualization are intentionally taken from relatively early in training. At this time, the agent has learned to only partially perceive the environment. Thus, we can visualize reasonable reconstructed scenes, but we have enough reconstruction error such that the perception component of our network will dominate the likelihood measurement we use for surprise. When the perception model improves after additional training, the relationship between intrinsic reward and image reconstruction error becomes less strong since intrinsic reward is also conditioned on the likelihood of the next state prediction which is determined by transition dynamics as well. We more clearly observe the correlation between reconstruction error and intrinsic reward in Figure 4 for a subset of the training samples in our model. The linear trend becomes weaker and rewards are not as tightly clustered for samples later in training which demonstrates that our model recognizes distinct transition dynamics based on likelihood. Through our analysis in Figure 5, we observe the success of our CVAE model in perceiving the game environment in Kung Fu Master. In addition, we validate the success of our designed relationship between intrinsic reward and ability to perceive the environment with this analysis in Figure 4. To execute our experiments, we leveraged the implementation of the PPO algorithm provided by. We used the Atari simulation environment for our video game simulations available in OpenAI Gym and developed by. We also incorporated infrastructure code from along with their inverse model implementation which we use as an action-prediction network in our approach. We trained each method for 10 million time steps on each environment. Each time step is associated with processing one new frame from the environment. On a 2080 Ti NVIDIA GPU, training for 10 million time steps took approximately 7 hours. This training time is associated with processing about 425 frames per second. To tune hyperparameters for loss weights in our model, we used intrinsic reward only and swept a range of values between zero and one in a grid search for each weight. Thus, we analyzed how different weighting of the loss values induced agents to explore differently, and we choose the combination which yielded the maximum ing extrinsic reward realization through intrinsic reward training only. Once we had optimal loss weights, we then tuned for the intrinsic and extrinsic reward weight combination. We swept a range of values between zero and one this weight combination as well, and we again chose the values which yielded the highest extrinsic reward. We tried 3 different scales of latent space dimension before choosing the one which caused our perception model to perform the best. The hyperparameters we used in our experiments are listed in Table 3. Note that the intrinsic reward weight and extrinsic reward weight were the same for ICM and our method though we tuned for the weights separately for each model. Also, note we took all the hyperparameters required in ICM other than reward weighting from. Intrinsic and extrinsic reward weights were not provided for using ICM with PPO. The first row shows the game scene images fed into our perception model as states. The second row shows the reconstructed images produced by our perception model. The remaining rows list intrinsic reward and embedded image reconstruction error respectively for each image pair.

Represent the short summary for finding the corresponding scientific article (long)

We introduce a method for computing an intrinsic reward for curiosity using metrics derived from sampling a latent variable model used to estimate dynamics.

We leverage deterministic autoencoders as generative models by proposing mixing functions which combine hidden states from pairs of images. These mixes are made to look realistic through an adversarial framework.