Question: Represent the article for finding a single-sentence TLDR

Question: Our work offers a new method for domain translation from semantic label maps and Computer Graphic (CG) simulation edge map images to photo-realistic im- ages. We train a Generative Adversarial Network (GAN) in a conditional way to generate a photo-realistic version of a given CG scene. Existing architectures of GANs still lack the photo-realism capabilities needed to train DNNs for computer vision tasks, we address this issue by embedding edge maps, and training it in an adversarial mode. We also offer an extension to our model that uses our GAN architecture to create visually appealing and temporally coherent videos. The topic of image to image translation and more generally video to video translation is of major importance for training autonomous systems. It is beneficial to train an autonomous agent in real environments, but not practical, since enough data cannot be gathered. However, using simulated scenes for training might lack details since a synthetic image will not be photorealistic and will lack the variability and randomness of real images, causing training to succeed up to a certain point. This gap is also referred to as the reality gap. By combining a non photo-realistic, simulated model with an available dataset, we can generate diverse scenes containing numerous types of objects, lightning conditions, colorization etc.. In this paper, we depict a new approach to generate images from a semantic label map and a flexible Deep Convolution Neural Network (DCNN) we called Deep Neural Edge Detector (DNED) which embed edge maps. we combine embedded edge maps which act as a skeleton with a semantic map as input to our model (fig 2. 1), The model outputs a photo-realistic version of that scene. Using the skeleton by itself will generate images that lack variability as it restricts the representation to that specific skeleton itself. Instead, we learn to represent skeletons by a neural network and at test time, we sample the closest appropriate skeleton the network has seen at training. Moreover, we have extended this idea to generate photo-realistic videos (i.e. sequence of images) with a novel loss that uses the optical flow algorithm for pixel coherency between consecutive images. Figure 1: in this paper we propose a method for generating photo-realistic images from semantic labels of a simulator scene. This figure provides images related to the Synthia dataset. Left -semantic map of the scene. Middle -generated image from pix2pixHD Wang et al. (2018b). Right -Our generated image. The texture and color space in our generated image is more natural giving the image the desired photo-realism. used as a super resolution generator. L1 loss for image generation is known to generate low quality images as the generated images are blurred and lack details. , are using a modified version of the perceptual loss, allowing generation of finer details in an image. Pix2pixHD Wang et al. (2018b) and are using a perceptual loss as well for training their networks, e.g.. Moreover, pix2pixHD are using instance maps as well as label maps to enable the generator to separate several objects of the same semantics. This is of high importance when synthesizing images having many instances of the same semantics in a single frame. As for video generation the loss used by Wang et al. (2018a), tend to be computationally expensive while our approach is simpler. We are using two generators of the same architecture, and they are mutually trained using our new optical flow based loss that is fed by dense optical flow estimation. Our evaluation method is and as it is a common metric being used for image and video generation schemes. We call this work s-Flow GAN since we embed Spatial information obtained from dense optical flow in a neural network as a prior for image generation and flow maps for video coherency. This optical flow is available since the simulated image is accessible at test time in the case of CG2real scheme. We make Three major contributions: First, our model can generate visually appealing photorealistic images from semantic maps having high definition details. Second, we incorporate a neural network to embed edge maps, thus allowing generation of diverse versions of the same scenes. Third, we offer a new loss function for generating natural looking videos using the above mentioned image generation scheme. please refer to this link for videos and comparison to related work. Generative Adversarial Networks (GAN) were introduced in 2014. This method generate images that look authentic to human observers. They do so by having two neural networks, one generating candidates while the other acts as a critique and tries to evaluate the generation quality , , , ,. GANs are widely used for image generation; some image synthesis schemes are used to generate low resolution images e.g. 32x32 while were able to generate higher resolution images (up to 512x512). In addition, Wang et al. (2018b) were able to generate even higher resolution images using coarse-to-fine generators. The reason generating high resolution images is challenging is the high dimensionality of the image generation task and the need to provide queues for high resolution ,. We offer queues as an edge map skeletons generated by our proposed DNED module. During training the DNED is trained to learn the representations of real image edge maps. During test the DNED is shown a CG (Computer Graphics) edge map, finds its best representation and provides the generator with an appropriate generated edge map sampled from real image edge maps distribution. Middle is the edge map extracted from the real image. Right is the real image being used by the discriminator for adversarial training. Please note that the real image is not used by the generator neither at training nor at test time, but only its edge map. The main issue in the CG2real model compared to the image to image models is that the simulators image is available to the generator at test time. We thus use the simulators image to extract the edge map allowing the generator to generate the necessary fine details in the output image. In the pix2pix setting, they used a , where the networks input is a semantic map of the scene, and while training in adversarial mode, a fake version of the real image is given to the discriminator to distinguish. In the CG2real setting in addition to the semantic map we also have access to the simulated image. Using the CG image as is, might be counter productive since it will be trained to reconstruct CG images and not photo-realistic ones. Conversely some of the underlying CG information correlates with the real world and can provide meaningful prior to the synthesis. Since the relevant information lies in the image high frequencies , we learn the distribution of edge maps in real images (high resolution details), and provide representation of it to the generator at test time. Some image generation tasks use label maps only, e.g.. The label maps provide only information about the class of a given pixel. In order to generate photo-realistic images, some use instance maps as well Wang et al. (2018b), This way, they can differentiate several adjacent objects of the same class. Nonetheless, while most datasets provide object level information about classes like cars, pedestrians, etc. they do not provide that information about vegetation and buildings. As a , the generated images might not correctly separate those adjacent objects, thus degrading photo-realism. Generating edge maps using neural networks is a well established method. Holistically-Nested Edge Detection (HED) provides holistic image training and prediction for multi-scale and multilevel feature learning. They use a composition of generated edge maps to learn a fine description of the edge scene. Inspired by their work, we train a neural network to learn edge maps of real images. As mentioned before, our generator requires an edge map as input. we get the edge map using a spacial Laplacian operator with threshold. Providing the generator with deterministic edge map will produce the same scene, so we train the DNED to take as input that deterministic edge map, learn its representation and produce a variant of that edge map, as a superposition of edges seen in real datasets. This way the generator will be able to produce a varaiaty of photorealistic images for the same scene. Since our approach (using edge maps) is not class dependent, we do not need instance map information to generate several adjacent instances of the same semantics. Moreover, this approach addresses the problem of generating fine details within a class like buildings and vegetation as can bee seen in fig 3. 2. Generating temporally coherent image sequences is a known challenge. Recent works use GANs to generate videos in an unconditional setting , , , by sampling from a random vector, but don't provide the generator with temporal constrains, thus generating non coherent sequences of images. Other works like video matting and video inpainting translate videos to videos but rely on problem specific constrains and designs. A recent work named vid2vid Wang et al. (2018a) offers to conditionally generate video from video and is considered to one of the best approaches to date. Using FlowNet 2.0 they predict the optical flow of the next image. In addition, they use a mask to differentiate between two parts; the hallucinated image generated from instance-level semantic segmentation masks and the predicted image from the previous frame. By adding these two parts, this method can combine the predicted details from the previously generated image, with the details from the newly generated image. Inspired by Wang et al. (2018a), we are using flow maps of consecutive images to generate temporally coherent videos. Contrary to Wang et al. (2018a) we are not using a CNN to predict the flow maps or a sequence generator, but a classical Computer vision approach. This is since a pre-trained network (trained on real datasets) failed to generalize and infer on simulated datasets e.g. Synthia. This enables better temporal coherency and improve video generation robustness. Our CG2real model aims to learn the conditional distribution of an image given a semantic map. Our video generation model aims to use this learned distribution for generating temporally coherent videos using the generated images from the CG2real scheme. We first depict the image generation scheme, then we review our video generation model. We use a conditional GAN to generate images from semantic maps as in. In order to generate images, the generator receives the semantic segmentation images s i and maps it to photo-realistic images x i. In parallel, the discriminator takes two images, The real image x i (ground truth) and the generated image f i and learns to distinguish between them. This supervised learning scheme is trained in the well-known min max game ,: In order to generate photo-realistic visually appealing images containing fine details, we provide a learnt representation of an edge map to the generator (fig 2.1), allowing it to learn the conditional distribution of real images given semantic maps and edge maps, i.e.: During training, given an example image x i, we can estimate its edge map by the well-known spatial Laplacian operator ,. This edge map is concatenated to the semantic label map and both are given as priors to the generator for adversarial training of the fake image f i vs. the real image x i. To allow a stable training we begin training our GAN with the edge maps from the Laplacian operator. After stabilization of the generator and discriminator, we provide our generator with edge maps from the DNED. We then jointly train the GAN with the DNED. The DNED architecture is a modified version of. In HED, they generate several sized versions of the edge map, each having a different receptive field. The purpose is to create an ensemble of edge maps, each allowing different level of details in the image. When superimposing all, the ing edge map will have coarse-to-fine level of details in the generated edge map image. By changing the weights of that ensemble, we can generate the desired variability in the generated edge map, thus allowing us to generate diverse versions of the output. To conclude, the loss function for training the DNED is: Where: d i (x), i = 0: 5 is the i th side output of a single scale, E(x) is the classic edge map generated by the spatial Laplacian operator, BCE is the binary cross entropy loss. N = 6 in our case. a i is the contribution of the i th scale to the ensemble. Increasing the resolution of the image might be challenging for GAN training. In other methods the discriminator needs a large receptive field , , , , requiring a deeper network or larger convolution kernels. Using a deeper network is prone to overfitting and in the case of GAN training, and might cause training to be unstably. This challenge is usually addressed by the multi-scale approach , , , ,. Since the DNED embed a learnt representation of skeletons, our architecture performs very well on higher resolution images. Our original generated images were of size [512x256]. We have successfully trained our model to generate images of size [768x384], i.e. 1.5 times larger in each dimension without changing the model while using a single discriminator (see 3.2). Figure 3: comparison of 768x384 pix images generated by pix2pixHD (Left) and our model (Right). Our model can generate lower level details in the image, thus improving its photo-realism. This figure provides an example comparing (768x384 pix) resolution images of pix2pixHD (left side) compared to our model (right). We showed that generating high quality images when using a single discriminator is feasible and training is stable. We provide comparison using our method with multi-scale discriminator 3.2. the FM loss is computed with k=1 for single layer discriminator and k=3 for multi layer one: Figure 4: Comparison of generated test images, when training with a single discriminator and a multi-scale one. The left image is generated when the generator was trained with a single discriminator, while the right image while using a multi-scale one. This figure demonstrates that when using our model (with our skeleton), training with a single discriminator might be enough. In addition, following , , , we are using the perceptual loss for improved visual performance and to encourage the discriminator distinguish real or fake samples using a pre traind. Where, P is the number of slices from a pre-trained VGG network and F L V GGi are the features extracted by the VGG network from the i th layer of the real and generated images respectively. To conclude, our overall objective for generating photo-realistic, diverse images in the CG2real setting is to minimize L CG2real: Figure 5: By using edge maps, the model learns to separate objects of the same semantics. The most dominant example is buildings. Unlike cars, pedestrians or bicycle riders, that are separable using the instance map, buildings are not. The semantic label provides the pixels in which the building exists. Considering the fact that a scene of adjacent buildings is somewhat common, the ability to separate them is of high value. Left -the label map. Middle -generated image by Wang et al. (2018b). Right -our generated image. Our model can generate unique adjacent buildings from the semantic label maps of better quality compared to Wang et al. (2018b). Figure 6: previous work test images Wang et al. (2018b) (Top) compared to our model test images (Bottom). The images generated by our model contain low level details, allowing the desired photorealism 3.3 VIDEO GENERATION Using pre trained CG2real networks, we generate two consecutive images, and then estimate two flow maps. The first flow map is between x i, x i+1, where x i and x i+1 are two consecutive real images. The second flow map is between G(s i, e i), G(s i+1, e i+1), where G(s i, e i) and G(s i+1, e i+1) are two consecutive generated (fake) images. Note that the generation of G(s i, e i), G(s i+1, e i+1) is done independently, meaning we apply our CG2real method twice, without any modifications. To conclude we enforce temporal coherency by using the following loss: Where and F(*) is the optical flow operator. This formulation eliminates the need of using a sequential generator as in Wang et al. (2018a), allowing us not only using our image generation model twice, which adds more constrains to the video generation scheme, but also avoid errors accumulation arising from positive feedback by feeding a generated image to the generator, as can be seen in figure 3.3 and in this video. By adding L f low to the L CG2real loss, the network learns to generate G(s i+1, e i+1) taking the flow maps into account, thus generating temporally coherent images as depicted in 3.3. Figure 7: block diagram of the video generation model. Two identical CG2real models generate Fake image (t) and Fake image (t+1). The two consecutive fake images are fed to the flow-fake estimator, while two consecutive real images are fed to the flow-real estimator. Both real and fake flow maps are trained using L 1 (F real, F f ake) loss. This enables the pre-trained CG2real models to learn the required coherency for generating photo-realistic videos. Our goal is to generate photo-realistic images. In (fig 3. 2) we can find some examples from the CG2real image synthesis task, and in (fig 4) present consecutive images depicting the video to video synthesis. We use the same evaluation methods as used by previous image to image works,e.g. pix2pix Isola et al., pix2pixHD Wang et al. (2018b and others. The evaluation process consist of performing semantic segmentation with a pre-trained seamntic segmentation network on synthesized images produces by our model, then calculating the semantic pixel accuracy and the mean intersection over union (mIoU) over the classes in the dataset. As shown in tables 1, 2 bellow, our network outperforms previous works. The ground-truth are the pixel accuracy and mIoU when performing the same semantic segmentation with the real images (Oracle). Furthermore, to evaluate the image generation quality, we used another metric to evaluate distances between datasets called FID (Frchet Inception Distance) ,. It is a very common metric for generative models as it correlates well with the visual quality of generated samples Wang et al. (2018a). FID calculates the distance between two multivariate Gaussians real and generated respectively; where X r ∼ N (µ r, Σ r) and X g ∼ N (µ g, Σ g) are the 2048-dimensional activations of the Inception-v3 pool3 layer , and ) is the score for image distributions X r and X g. Lower FID score is better, meaning higher similarity between real and generated samples. As can be seen in tables 1, 2, pix2pixHDs are better than pix2pix for pixel accuracy and mIoU. Our are better than pix2pixHD, and almost meet the oracles on both and cityscapes.. FVD is a metric for video generation models evaluation and it uses a modified version of FID. we calculated the FVD score for our generated video (Ours-vid) w.r.t. the Oracle (real video) and did the same for vid2vid w.r.t the same Oracle. Our FVD score on the video test set is 0.326 while vid2vid's is 0.706 meaning our videos are more than twice similar to the oracle. we suggest that this substantial margin stems from the errors accumulated in the video generation model of vid2vid (fig 4). As mentioned, Our video generation model uses our flow loss therefore does not encounter this phenomena. Figure 8: Comparison of video generation. Up -images generated by vid2vid Wang et al. (2018a). Down -images generated by our video generation model. Our generated images are temporally coherent and visually appealing In our images sky is more natural, road signs are clearer and buildings have finer level of details. This example emphasizes the error propagation of vid2vid's model wile our model does not accumulate errors (see street lights in upper right corner of each image). The main objective of the video generation model is to enable generating non flickering images by giving objects in consecutive images the same color and texture, i.e. sample from the same area in the latent spaces. full videos can be seen here. We present a CG2real conditional image generation as well as a conditional video synthesis. We offer to use a network learning the distribution of edge maps from real images and integrate it into a generator (DNED). We were able to generate highly detailed and diverse images thus enabling better photo-realism. Using the DNED enable generating diverse yet photo-realistic realizations of the same desired scene without using instance maps. As for video generation, we offer a new scheme that utilizes flow maps allowing better temporal coherence in videos. We compared our model to recent works and found that it outperforms both current quantitative and more importantly generates appealing images. Furthermore, our video generation model generates temporally coherent and consistent videos. A APPENDIX Figure 9: Additional Test images from Cityscapes Dataset. Left -pix2pixHD. Right -Ours. These images further demonstrate the photo-realism achieved by our model. Figure 10: Additional test images on Synthia dataset. Left -pix2pixHD. Right -Ours. These images demonstrate improved image quality, better and finer details in the generated objects, buildings and vegetation. Figure 11: Additional test images on Synthia dataset. Left -pix2pixHD. Right -Ours. These images demonstrate improved image quality, better and finer details in the generated objects, buildings and vegetation. Figure 12: Test video on CityScapes. Left -vid2vid. Right -Ours video gen model. These images demonstrate better temporal coherency in the generated images. Moreover, in the top left corner of the left video, we notice the error propagates. Better yet, the buildings in the right video are more reasonable, w.r.t. windows, shades, general texture, etc. As for image quality, the road signs in the right video are better emphasized.

Represent the TLDR for finding the corresponding scientific article:

Simulation to real images translation and video generation

Representing melodies as images with semantic units aligned we can generate them using a DCGAN without any recurrent components.


Question: Represent the scientific article to find a short TLDR

Question: Deep neural networks are widely used in various domains, but the prohibitive computational complexity prevents their deployment on mobile devices. Numerous model compression algorithms have been proposed, however, it is often difficult and time-consuming to choose proper hyper-parameters to obtain an efficient compressed model. In this paper, we propose an automated framework for model compression and acceleration, namely PocketFlow. This is an easy-to-use toolkit that integrates a series of model compression algorithms and embeds a hyper-parameter optimization module to automatically search for the optimal combination of hyper-parameters. Furthermore, the compressed model can be converted into the TensorFlow Lite format and easily deployed on mobile devices to speed-up the inference. PocketFlow is now open-source and publicly available at https://github.com/Tencent/PocketFlow. Deep learning has been widely used in various areas, such as computer vision, speech recognition, and natural language translation. However, deep learning models are often computational expensive, which limits further applications on mobile devices with limited computational resources. To address this dilemma between accuracy and computational complexity, numerous algorithms have been proposed to compress and accelerate deep networks with minimal performance degradation. Commonly-used approaches include low-rank decomposition BID15 BID14, channel pruning (a.k.a. structured pruning) BID6 BID17, weight sparsification (a.k.a. non-structured pruning) BID16, and weight quantization BID1 BID2. However, these algorithms usually involve several hyper-parameters that may have a large impact on the compressed model's performance. It can be quite difficult to efficiently choose proper hyper-parameter combinations for different models and learning tasks. Recently, some researches adopted reinforcement learning methods to automatically determine hyperparameters for channel pruning BID4 and weight sparsification BID5 algorithms. In this paper, we present an automated framework for compressing and accelerating deep neural networks, namely PocketFlow. We aim at providing an easy-to-use toolkit for developers to improve the inference efficiency with little or no performance degradation. PocketFlow has inte-grated a series of model compression algorithms, including structured/non-structured pruning and uniform/non-uniform quantization. A hyper-parameter optimizer is incorporated to automatically determine hyper-parameters for model compression components. After iteratively training candidate compressed models and adjusting hyper-parameters, a final compressed model is obtained to maximally satisfy user's requirements on compression and/or acceleration ratios. The ing model can be exported as a TensorFlow-Lite file for efficient deployment on mobile devices. The proposed framework mainly consists of two categories of algorithm components, i.e. learners and hyper-parameter optimizers, as depicted in FIG0. Given an uncompressed original model, the learner module generates a candidate compressed model using some randomly chosen hyperparameter combination. The candidate model's accuracy and computation efficiency is then evaluated and used by hyper-parameter optimizer module as the feedback signal to determine the next hyper-parameter combination to be explored by the learner module. After a few iterations, the best one of all the candidate models is output as the final compressed model. A learner refers to some model compression algorithm augmented with several training techniques as shown in FIG0. Below is a list of model compression algorithms supported in PocketFlow: NonUniformQuantLearner weight quantization with non-uniform reconstruction levels BID2 All the above model compression algorithms can trained with fast fine-tuning, which is to directly derive a compressed model from the original one by applying either pruning masks or quantization functions. The ing model can be fine-tuned with a few iterations to recover the accuracy to some extent. Alternatively, the compressed model can be re-trained with the full training data, which leads to higher accuracy but usually takes longer to complete. To further reduce the compressed model's performance degradation, we adopt network distillation to augment its training process with an extra loss term, using the original uncompressed model's outputs as soft labels. Additionally, multi-GPU distributed training is enabled for all learners to speed-up the time-consuming training process. For model compression algorithms, there are several hyper-parameters that may have a large impact on the final compressed model's performance. It can be quite difficult to manually determine proper values for these hyper-parameters, especially for developers that are not very familiar with algorithm details. Therefore, we introduce the hyper-parameter optimizer module to iteratively search for the optimal hyper-parameter setting. In PocketFlow, we provide several implementations of hyper-parameter optimizer, based on models including Gaussian Processes (GP) BID11, Tree-structured Parzen Estimator (TPE) BID0, and Deterministic Deep Policy Gradients (DDPG) BID10. The hyper-parameter setting is optimized through an iterative process. In each iteration, the hyper-parameter optimizer chooses a combination of hyperparameter values, and the learner generates a candidate model with fast fast-tuning. The candidate model is evaluated to calculate the reward of the current hyper-parameter setting. After that, the hyper-parameter optimizer updates its model to improve its estimation on the hyper-parameter space. Finally, when the best candidate model (and corresponding hyper-parameter setting) is selected after some iterations, this model can be re-trained with full data to further reduce the performance loss. For empirical evaluation, we adopt PocketFlow to compress and accelerate classification models on the CIFAR-10 BID9 and ILSVRC-12 BID12 data sets. In FIG1, we use ChannelPrunedLearner to speed-up ResNet-56 BID3 to reduce its computational complexity. We observe that the accuracy loss under 2.5× acceleration is 0.4% and under 3.3× acceleration is 0.7%, and compressed models are more efficient and effective that the shallower ResNet-44 model. In FIG1, we use WeightSparseLearner to compress MobileNet BID7 to reduce its model size. We discover that the compressed model achieves similar classification accuracy with much smaller model size than MobileNet, Inception-v1 BID13, and ResNet-18 models. The compressed models generated by PocketFlow can be exported as TensorFlow Lite models and directly deployed on mobile devices using the mobile-optimized interpreter. In TAB2, we compare the classification accuracy, model size, and inference latency 1 of original and compressed models. With ChannelPrunedLearner, the compressed model achieves 1.53× speed-up with 2.0% loss in the top-5 classification accuracy. With UniformQuantLearner, we achieve 2.46× speed-up after applying 8-bit quantization on the MobileNet model, while the top-5 accuracy loss is merely 0.6%. In this paper, we present the PocketFlow framework to boost the deployment of deep learning models on mobile devices. Various model compression algorithms are integrated and hyper-parameter optimizers are introduced into the training process to automatically generate highly-accurate compressed models with minimal human effort.

Represent the short summary for finding the corresponding scientific article (long)

We propose PocketFlow, an automated framework for model compression and acceleration, to facilitate deep learning models' deployment on mobile devices.

represent this text to find a much much longer article about it with say hundreds-thousands of words

We propose accelerating Batch Normalization (BN) through sampling less correlated data for reduction operations  with regular execution pattern, which achieves up to 2x and 20% speedup for BN itself and the overall training, respectively.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Generative models provide a way to model structure in complex distributions and have been shown to be useful for many tasks of practical interest. However, current techniques for training generative models require access to fully-observed samples. In many settings, it is expensive or even impossible to obtain fully-observed samples, but economical to obtain partial, noisy observations. We consider the task of learning an implicit generative model given only lossy measurements of samples from the distribution of interest. We show that the true underlying distribution can be provably recovered even in the presence of per-sample information loss for a class of measurement models. Based on this, we propose a new method of training Generative Adversarial Networks (GANs) which we call AmbientGAN. On three benchmark datasets, and for various measurement models, we demonstrate substantial qualitative and quantitative improvements. Generative models trained with our method can obtain $2$-$4$x higher inception scores than the baselines. The output of the generator is passed through a simulated random measurement function f Θ. The discriminator must decide if a measurement is real or generated. models of the data structure. Recent work has shown that generative models can be particularly effective for easier sensing BID4; BID20 ]-but if sensing is expensive in the first place, how can we collect enough data to train a generative model to start with?This work solves this chicken-and-egg problem by training a generative model directly from noisy or incomplete samples. We show that our observations can be even projections or more general measurements of different types and the unknown distribution is still provably recoverable. A critical assumption for our framework and theory to work is that the measurement process is known and satisfies certain technical conditions. We present several measurement processes for which it is possible to learn a generative model from a dataset of measured samples, both in theory and in practice. Our approach uses a new way of training GANs, which we call AmbientGAN. The idea is simple: rather than distinguish a real image from a generated image as in a traditional GAN, our discriminator must distinguish a real measurement from a simulated measurement of a generated image; see FIG0. We empirically demonstrate the effectiveness of our approach on three datasets and a variety of measurement models. Our method is able to construct good generative models from extremely noisy observations and even from low dimensional projections with drastic per-sample information loss. We show this qualitatively by exhibiting samples with good visual quality, and quantitatively by comparing inception scores BID28 ] to baseline methods. Theoretical . We first consider measurements that are noisy, blurred versions of the desired images. That is, we consider convolving the original image with a Gaussian kernel and adding independent Gaussian noise to each pixel (our actual theorem applies to more general kernels and noise distributions). Because of the noise, this process is not invertible for a single image. However, we show that the distribution of measured images uniquely determines the distribution of original images. This implies that a pure Nash equilibrium for the GAN game must find a generative model that matches the true distribution. We show similar for a dropout measurement model, where each pixel is set to zero with some probability p, and a random projection measurement model, where we observe the inner product of the image with a random Gaussian vector. Empirical . Our empirical work also considers measurement models for which we do not have provable guarantees. We present on some of our models now and defer the full exploration to Section 8.In FIG1, we consider the celebA dataset of celebrity faces BID19 ] under randomly placed occlusions, where a randomly placed square containing 1/4 of the pixels is set to zero. It is hard to inpaint individual images, so cleaning up the data by inpainting and then learning a GAN on the yields significant artifacts. By incorporating the measurement process into the GAN training, we can produce much better samples. In FIG2 we consider learning from noisy, blurred version of images from the celebA dataset. Each image is convolved with a Gaussian kernel and then IID Gaussian noise is added to each pixel. Learning a GAN on images denoised by Wiener deconvolution leads to poor sample quality while our models are able to produce cleaner samples. In FIG2, we consider learning a generative model on the 2D images in the MNIST handwritten digit dataset BID17 ] from pairs of 1D projections. That is, measurements consist of picking two random lines and projecting the image onto each line, so the observed value along the line is the sum of all pixels that project to that point. We consider two variants: in the first, the choice of line is forgotten, while in the second the measurement includes the choice of line. We find for both variants that AmbientGAN recovers a lot of the underlying structure, although the first variant cannot identify the distribution up to rotation or reflection. There are two distinct approaches to constructing neural network based implicit generative models; autoregressive BID15 BID25 a) ], and adversarial BID11 ]. Some combination approaches have also been successful BID21 ].The adversarial framework has been shown to be extremely powerful in modeling complex data distributions such as images BID27;; BID3 ], video BID18; BID31 ], and 3D models BID0; BID32. A learned generative model can be useful for many applications. A string of papers BID4 BID14; BID33 ] explore the utility of generative priors to solve ill-posed inverse problems. BID29 ] demonstrate that synthetic data can be made more realistic using GANs. BID14 ] and [] show how to translate images from one domain to another using GANs. The idea of operating generators and discriminators on different spaces has been proposed before. BID22 ] explores an interesting connection of training stability with low dimensional projections of samples. They show that training a generator against an array of discriminators, each operating on a different low-dimensional projection of the data can improve stability. Our work is also closely related to BID10 ] where the authors create 3D object shapes from a dataset of 2D projections. We note that their setup is a special case of the AmbientGAN framework where the measurement process creates 2D projections using weighted sums of voxel occupancies. Throughout, we use superscript'r' to denote real or true distribution, superscript'g' for the generated distributions,'x' for the underlying space and'y' for measurements. Let p r x be a real underlying distribution over R n. We observe lossy measurements performed on samples from p r x. If we let m be the size of each observed measurement, then, each measurement is an output of some measurement function f θ: R n → R m, parameterized by θ. We allow the measurement function to be stochastic by letting the parameters of the measurement functions have a distribution p θ. With this notation, for a given x and θ, the measurements are given by y = f θ (x). We assume that it is easy to sample Θ ∼ p θ and to compute f θ (x) for any x and θ. The distributions p r x and p θ naturally induce a distribution over the measurements y which we shall denote by p r y. In other words, if X ∼ p r x and DISPLAYFORM0 Our task is the following: there is some unknown distribution p r x and a known distribution p θ. We are given a set of IID realizations {y 1, y 2, . . ., y s} from the distribution p r y. Using these, our goal is to create an implicit generative model of p r x, i.e., a stochastic procedure that can sample from p r x. Our main idea is to combine the measurement process with the adversarial training framework, as shown in FIG0. Just like in the standard GAN setting, let Z ∈ R k, Z ∼ p z be a random latent vector for a distribution p z that is easy to sample from, such as IID Gaussian or IID uniform. Let DISPLAYFORM1, and let p g x be the distribution of X g. Thus, our goal is to learn a generator G such that p g x is close to p r x. However, unlike the standard GAN setting, we do not have access to the desired objects (X ∼ p r x). Instead, we only have a dataset of measurements (samples from Y ∼ p r y). Our main idea is to simulate random measurements on the generated objects X g, and use the discriminator to distinguish real measurements from fake measurements. Thus, we sample a random measurement function f Θ by sampling Θ ∼ p θ and apply it on X g to obtain DISPLAYFORM2 We set up the discriminator to predict if a given y is a sample from the real measurement distribution p r y as opposed to the generated measurement distribution p g y. Thus, the discriminator is a function D: R m → R.We let q(·) be the quality function that is used to define the objective, based on the discriminator output. For vanilla GAN, q(x) = log(x) and for Wasserstein GAN ], q(x) = x. Accordingly, the AmbientGAN objective is the following: DISPLAYFORM3 We additionally require f θ to be differentiable with respect to its inputs for all θ. We implement G and D as feedforward neural networks. With these assumptions, our model is end-to-end differentiable and can be trained using an approach similar to the standard gradient-based GAN training procedure. In each iteration, we sample Z ∼ p z, Θ ∼ p θ, and Y r ∼ UNIF{y 1, y 2, . . ., y s} to use them to compute stochastic gradients of the objective with respect to parameters in G and D by backpropagation. We alternate between updates to parameters of D and updates to parameters of G.We note that our approach is compatible with and complementary to the various improvements proposed to the GAN objective, network architectures, and the training procedures. Additionally, we can easily incorporate additional information, such as per sample labels, in our framework through conditional versions of the generator and discriminator. This is exemplified in our experiments, where we use unconditional and conditional versions of DCGAN BID27 ], unconditional Wasserstein GAN with gradient penalty BID12 ], and an Auxiliary Classifier Wasserstein GAN BID23 ] with gradient penalty. Now, we describe the measurement models that we use for our theoretical and empirical . We primarily focus on 2D images and thus our measurement models are tailored to this setting. The AmbientGAN learning framework, however, is more general and can be used for other data formats and other measurement models as well. For the rest of this section, we assume that input to the measurement function (x) is a 2D image. We consider the following measurement models:Block-Pixels: Each pixel is independently set to zero with probability p. Convolve+Noise: Let k be a convolution kernel and let Θ ∼ p θ be the distribution of noise. Then the measurements are given by f Θ (x) = k * x + Θ, where * is the convolution operator. Block-Patch: A randomly chosen k × k patch is set to zero. Keep-Patch: All pixels outside a randomly chosen k × k patch are set to zero. Extract-Patch: A random k ×k patch is extracted. Note that unlike the previous measurement function, the information about the location of the patch is lost. Pad-Rotate-Project: We pad the image on all four sides by zeros. Then we rotate the image by a random angle (θ) about its center. The padding is done to make sure that the original pixels stay within the boundary. Finally, for each channel in the image, we sum the pixels along the vertical axis to get one measurement vector. PadRotate-Project-θ: This is the same as the previous measurement function, except that along with the projection values, the chosen angle is also included in the measurements. Gaussian-Projection: We project onto a random Gaussian vector which is included in the measurements. So, Θ ∼ N (0, I n), and f Θ (x) = (Θ, Θ, x). We show that we can provably recover the true underlying distribution p r x for certain measurement models. Our broad approach is to show that there is a unique distribution p r x consistent with the observed measurement distribution p r y, i.e., the mapping of distributions of samples p r x to distribution of measurements p r y is invertible even though the map from an individual image x to its measurements f θ (x) is not. If this holds, then the following lemma immediately gives a consistency guarantee with the AmbientGAN training procedure. Lemma 5.1. As in Section 3, let p r x be the data distribution, p θ be the distribution over parameters of the measurement functions and p r y be the induced measurement distribution. Further, assume that for the given p θ, there is a unique probability distribution p r x that induces the given measurement distribution p r y. Then, for the vanilla GAN model BID11 DISPLAYFORM0 All proofs including this one are deferred to Appendix A. Note that the previous lemma makes a non-trivial assumption of uniqueness of the true underlying distribution given the measurement distribution. The next few theorems show that this assumption is satisfied under Gaussian-Projection, Convolve+Noise and Block-Pixels measurement models, thus showing that that we can recover the true underlying distribution with the AmbientGAN framework. We remark that the required conditions in the preceding theorem are easily satisfied for the common setting of Gaussian blurring kernel with additive Gaussian noise. The same guarantee can be generalized for any continuous and invertible function instead of a convolution. We omit the details. Our next theorem makes an assumption of a finite discrete set of pixel values. This assumption holds in most practical scenarios since images are represented with a finite number of discrete values per channel. In this setting, in addition to a consistency guarantee, we also give a sample complexity for approximately learning the distributions in the AmbientGAN framework. Theorem 5.4. Assume that each image pixel takes values in a finite set P. Thus x ∈ P n ⊂ R n. Assume 0 ∈ P, and consider the Block-Pixels measurement model (Section 4) with p being the probability of blocking a pixel. If p < 1, then there is a unique distribution p r x that can induce the measurement distribution p r y. Further, for any > 0, δ ∈, given a dataset of DISPLAYFORM1 IID measurement samples from p r y, if the discriminator D is optimal, then with probability ≥ 1 − δ over the dataset, any optimal generator G must satisfy DISPLAYFORM2 We used three datasets for our experiments. MNIST is a dataset of 28 × 28 images of handwritten digits BID17 ]. CelebA is a dataset of face images of celebrities BID19 ]. We use an aligned and cropped version where each image is 64 × 64 RGB. The CIFAR-10 dataset consists of 32 × 32 RGB images from 10 different classes BID16 DISPLAYFORM0 We briefly describe the generative models we used for our experiments. More details on architectures and hyperparameters can be found in the appendix. For the MNIST dataset, we use two GAN models. The first model is a conditional DCGAN which follows the architecture in [] 1, while the second model is an unconditional Wasserstein GAN with gradient penalty (WGANGP) which follows the architecture in BID12 2. For the celebA dataset, we use an unconditional DCGAN and follow the architecture in BID27 3. For the CIFAR-10 dataset, we use an Auxiliary Classifier Wasserstein GAN with gradient penalty (ACWGANGP) which follows the residual architecture in BID12 4.For measurements with 2D outputs, i.e. Block-Pixels, Block-Patch, Keep-Patch, Extract-Patch, and Convolve+Noise (see Section 4), we use the same discriminator architectures as in the original work. For 1D projections, i.e. Pad-Rotate-Project, Pad-Rotate-Project-θ, we use fully connected discriminators. The architecture of the fully connected discriminator used for the MNIST dataset was 25-25-1 and for the celebA dataset was 100-100-1. Now, we describe some baseline approaches that we implemented to evaluate the relative performance of the AmbientGAN framework. Recall that we have a dataset of IID samples {y 1, y 2, . . . y s} from the measurement distribution p r y and our goal is to create an implicit generative model for p r x. A crude baseline is to ignore that any measurement happened at all. In other words, for cases where the measurements lie in the same space as the full-samples (for example Convolve+Noise) we can learn a generative model directly on the measurements and test how well it approximates the true distribution p r x. We call this the "ignore" baseline. A stronger baseline is based on the following observation: If the measurement functions f θ were invertible, and we observed θ i for each measurement y i in our dataset, we could just invert the functions to obtain full-samples DISPLAYFORM0 θi (y i). Then we could directly learn a generative model using these full-samples. Notice that both assumptions are violated in the AmbientGAN setting. First, we may not observe θ i and second, the functions may not be invertible. Indeed all the measurement models in Section 4 violate one of the assumptions. However, we can try to approximate an inverse function and use the inverted samples to train a generative model. Thus, given a measurement y i = f θi (x i), we try to "unmeasure" it and obtain x i, an estimate of x i. We then learn a generative model with the estimated inverse samples and test how well it approximates p For the measurement models described in Section 4, we now describe the methods we used to obtain approximate inverse functions: (a) For the Block-Pixels measurements, a simple approximate inverse function is to just blur the image so that zero pixels are filled in from the surrounding. We also implemented a more sophisticated approach to fill in the pixels by using total variation inpainting. (b) For Convolve+Noise measurements with a Gaussian kernel and additive Gaussian Noise, we approximate the inverse by a Wiener deconvolution. (c) For Block-Patch measurements, we use the Navier Stokes based inpainting method BID2 ] to fill in the zero pixels. For other measurement models, it is unclear how to obtain an approximate inverse function. For the Keep-Patch measurement model, no pixels outside a box are known and thus inpainting methods are not suitable. Inverting Extract-Patch measurements is even harder since the information about the position of the patch is also lost. For the Pad-Rotate-Project-θ measurements, a conventional technique is to sample many angles, and use techniques for inverting the Radon transform BID7 ]. However, since we observe only a few projections at a time, these methods aren't readily applicable. Inverting Pad-Rotate-Project measurements is even harder since it lacks information about θ. So, on this subset of experiments, we report only the with the AmbientGAN models. We present some samples generated by the baselines and our models. For each experiment, we show the samples from the dataset of measurements (Y r) available for training, samples generated by the baselines (when applicable) and the samples generated by our models (X g). We show samples only for a selected value of parameter settings. More are provided in the appendix. All on MNIST are deferred to the appendix. Block-Pixels: FIG5 shows on celebA with DCGAN and FIG7 on CIFAR-10 with ACW-GANGP. We see that the samples are heavily degraded in our measurement process (left image). Thus, it is challenging for baselines to invert the measurements process, and correspondingly, they do not produce good samples (middle image). Our models are able to produce images with good visual quality (right image).Convolve+Noise: We use a Gaussian kernel and IID Gaussian noise. FIG2 shows on celebA with DCGAN. We see that the measurements are drowned in noise (left image) and the baselines Block-Patch, Keep-Patch: FIG1 shows the for Block-Patch and FIG6 for Keep-Patch measurements on celebA with DCGAN. On both measurement distributions, our models are able to create coherent faces (right image) by observing only parts of one image at a time.1D projections: Pad-Rotate-Project and Pad-Rotate-Project-θ measurement models exhibit drastic signal degradation; most of the information in a sample is lost during the measurements process. For our experiments, we use two measurements at a time. FIG2 shows the on MNIST with DCGAN. While the first model is able to learn only up to rotation and reflection (left image), we note that generated digits have similar orientations and chirality within each class without any explicit incentive. We hypothesize that the model prefers this mode because it is easier to learn with consistent orientation per class. The second measurement model contains the rotation angle and thus produces upright digits (right image). While in both cases, the generated images are of lesser visual quality, our method demonstrates that we can produce images of digits given only 1D projections. Failure case: In FIG6, we show the samples obtained from our model trained on celebA dataset with Pad-Rotate-Project-θ measurements with a DCGAN. We see that the model has learned a very crude outline of a face, but lacks details. This highlights the difficulty in learning complex distributions with just 1D projections and a need for better understanding of distribution recovery under projection measurement model as well as better methods for training GANs. We report inception scores BID28 ] to quantify the quality of the generative models learned in the AmbientGAN framework. For the CIFAR-10 dataset, we use the Inception model BID30 ] trained on the ImageNet dataset BID8 1. For computing a similar score on MNIST, we trained a classification model with two conv+pool layers followed by two fully connected layers 2. The final test set accuracy of this model was 99.2%. For Block-Pixels measurements on MNIST, we trained several models with our approach and the baselines, each with a different probability p of blocking pixels. For each model, after convergence, we computed the inception score using the network described above. A plot of the inception scores as a function of p is shown in Fig. 7 (left). We note that at p = 0, i.e. if no pixels are blocked, our model is equivalent to a conventional GAN. As we increase p, the baseline models quickly start to perform poorly, while the AmbientGAN models continue to perform relatively well. For the Convolve+Noise measurements with a Gaussian kernel of radius 1 pixel, and additive Gaussian noise with zero mean and standard deviation σ, we trained several models on MNIST by varying the value of σ. A plot of the inception score as a function of σ is shown in Fig. 7 (right). We see that for small variance of additive noise, Wiener deconvolution and the "ignore" baseline perform quite well. However, as we start to increase the noise levels, these baselines quickly deteriorate in performance, while the AmbientGAN models maintain a high inception score. For 1D projection measurements, we report the inception scores for the samples produced by the AmbientGAN models trained with two projection measurements at a time. The Pad-Rotate-Project model produces digits at various orientations and thus does quite poorly, achieving an inception score of just 4.18. The model with Pad-Rotate-Project-θ measurements produces well-aligned digits and achieves an inception score of 8.12. For comparison, the vanilla GAN model trained with fullyobserved samples achieves an inception score of 8.99. Thus, the second model comes quite close to the performance of the fully-observed case while being trained only on 1D projections. In Fig. 8 (left), we show a plot of inception score vs the probability of blocking pixels p in the BlockPixels measurement model on CIFAR-10. We note that the total variation inpainting method is quite slow and the performance on MNIST was about the same as unmeasure-blur baseline. So, we do not run inpainting baselines on the CIFAR-10 dataset. From the plots, we see a trend similar to the plot obtained with MNIST (Fig. 7, left), showing the superiority of our approach over baselines. We show the inception score as a function of training iteration in Fig. 8 (right).Generative models are powerful tools, but constructing a generative model requires a large, highquality dataset of the distribution of interest. We show how to relax this requirement, by learning a distribution from a dataset that only contains incomplete, noisy measurements of the distribution. We hope that this will allow for the construction of new generative models of distributions for which no high-quality dataset exists. Lemma. As in Section 3, let p r x be the data distribution, p θ be the distribution over parameters of the measurement functions and p r y be the induced measurement distribution. Further, assume that for the given p θ, there is a unique probability distribution p r x that induces the given measurement distribution p r y. Then, for the vanilla GAN model BID11 ], if the Discriminator D is optimal, so that DISPLAYFORM0 Proof. From the same argument as in Theorem 1 in BID11 ], it follows that p g y = p r y. Then, since there is a unique probability distribution p Proof. We note that Since Θ ∼ N (0, I n), all possible directions for projections are covered. Further, since the measurement model includes the projection vector Θ as a part of the measurements, in order to match the measurement distribution, the underlying distribution p r x must be such that all 1D marginals are matched. Thus, by Cramer-Wold theorem BID6 ], any sequence of random vectors that match the 1D marginals must converge in distribution to the true underlying distribution. Thus, in particular, there is a unique probability distribution p r x that can match all 1D marginals obtained with the Gaussian projection measurements. Theorem. Let F(·) denote the Fourier transform and let supp(·) be the support of a function. Consider the Convolve+Noise measurement model (Section 4) with the convolution kernel k and additive noise distribution p θ. If supp(F(k)) c = φ and supp(F(p θ)) c = φ, then there is a unique distribution p r x that can induce the measurement distribution p r y. DISPLAYFORM0 With a slight abuse of notation, we will denote the probability density functions (pdf) also by p subscripted with the variable name. Then we have DISPLAYFORM1 where the penultimate step follows since by assumption, F(k) is nowhere 0. In the last step, F −1 is the inverse Fourier transform. Thus, there is a bijective map between X and Z. Since the Fourier and the inverse Fourier are continuous transformations, this map is also continuous. So, we can write Z = h(X), where h is a bijective, differentiable function. So, the pdfs of X and Z are related as DISPLAYFORM2 where J h (x) is the Jacobian of h evaluated atx. Now, note that since Y is a sum of two random variables, its pdf is a convolution of the individual probability density functions. So we have: DISPLAYFORM3 Taking the Fourier transform on both sides, we have DISPLAYFORM4 where the penultimate step follows since by assumption, F(p θ) is nowhere 0.Combining the two , we have a reverse map from the measurement distribution p y to a sample distribution p x. Thus, the reverse map uniquely determines the true underlying distribution p x, concluding the proof. We first state a slightly different version of Theorem 1 from BID11 ] for the discrete setting. We shall use [n] to denote the set {1, 2, . . . n}, and use I(·) to denote the indicator function. Lemma 10.1. Consider a dataset of measurement samples {y 1, y 2, . . . y s}, where each y i ∈ [t]. We define the empirical version of the vanilla GAN objective as DISPLAYFORM0 For j ∈ [t], letp r y (j) = I(y i = j)/s be the empirical distribution of samples. Then the optimal discriminator for the empirical objective is such that DISPLAYFORM1 Additionally, if we fix the discriminator to be optimal, then any optimal generator must satisfy p g y =p r y.Proof. The Empirical Risk Minimization (ERM) version of the loss is equivalent to the taking expectation of the data dependent term with respect to the empirical distribution. Replacing the real data distribution with the empirical version in the proof of Theorem 1 from BID11 ], we obtain the . Now we give a proof of Theorem 5.4.Theorem. Assume that each image pixel takes values in a finite set P. Thus x ∈ P n ⊂ R n. Assume 0 ∈ P, and consider the Block-Pixels measurement model (Section 4) with p being the probability of blocking a pixel. If p < 1, then there is a unique distribution p Proof. We first consider a more general case and apply that to the Block-Pixels model. Consider a discrete distribution p x over [t]. We apply random measurement functions to samples from p x to obtain measurements. Assume that each measurement also belongs to the same set, i.e. [t]. Let A ∈ R t×t be the transition matrix so that A ij is the probability (under the randomness in measurement functions) that measurement i was produced by sample j. Then the distribution over measurements p y can be written in terms of p x and A as: DISPLAYFORM2 Thus, if the matrix A is invertible, we can guarantee that the distribution p x is recoverable from p y.Assuming A is invertible, we now turn to the sample complexity. Let λ be the minimum of magnitude of eigenvalues of A. Since A is invertible, λ > 0. Let the dataset of measurements be {y 1, y 2, . . . y s}. For j ∈ [t] and for k ∈ [s], Let Y j k = I(y k = j). Then for any > 0, we have DISPLAYFORM3 where we used union bound and Chernoff inequalities. Setting this to δ, we get s = t 2 2λ 2 2 log 2t δ.From Lemma 10.1, we know that the optimal generator must satisfy p. Thus, we obtain that with probability ≥ 1 − δ, DISPLAYFORM4 =.Now we turn to the specific case of Block-Pixels measurement. We proceed by dividing the set of all possible |P | n images into n + 1 classes. The i-th class has those images that have exactly i pixels with zero value. We sort the images according to their class number (arbitrary ordering within the class) and consider the transition matrix A. Note that given an image from class i it must have j ≥ i zero pixels after the measurement. Also, no image in class i can produce another image in the same class after measurements. Thus, the transition matrix is lower triangular. Since each pixel is blocked independently with probability p and since there are n pixels, the event that no pixels are blocked occurs with probability FIG0 n. Thus, every image has at least (1 − p) n chance of being unaffected by the measurements. Any unaffected image maps to itself and thus forms diagonal entries in the transition matrix. So, we observe that the diagonal entries of the transition matrix are strictly positive and their minimum value is (1 − p) n.For a triangular matrix, the diagonal entries are precisely the eigenvalues and hence we have proved that A is invertible and the smallest eigenvalue is (1 − p) n. Combined with the above, by setting λ = (1 − p) n, and t = |P | n, we conclude the proof. The DCGAN model on MNIST follows the architecture in BID27 ]. The noise input to the generator (Z) has 100 dimensions where each coordinate is sampled IID Uniform on [−1, 1]. The generator uses two linear layers followed by two deconvolutional layers. The labels are concatenated with the inputs of each layer. The discriminator uses two convolutional layers followed by two linear layers. As with the generator, the labels are concatenated with the inputs of each layer. Batch-norm is used in both generator and the discriminator. The WGANGP model on MNIST follows the architecture in BID12 ]. The generator takes in a latent vector of 128 dimensions where each coordinate is sampled IID Uniform on [−1, 1]. The generator then applies one linear and three deconvolutional layers. The discriminator uses three convolutional layers followed by one linear layer. Batch-norm is not used. The unconditional DCGAN model on celebA follows the architecture in BID27 ]. The latent vector has 100 dimensions where each coordinate is Uniform on [−1, 1]. The generator applies one linear layer followed by four deconvolutional layers. The discriminator uses four convolutional layers followed by a linear layer. Batch-norm is used in both generator and the discriminator. The ACWGANGP model on CIFAR-10 follows the residual architecture in BID12 ]. The latent vector has 128 dimensions where each coordinate is sampled from IID standard Gaussian distribution. The generator has a linear layer followed by three residual blocks. Each residual block consists of two repetitions of the following three operations: conditional batch normalization followed by a nonlinearity followed by an upconvolution layer. The residual blocks are followed by another conditional batch normalization, a final convolution, and a final tanh non-linearity. The discriminator consists of one residual block with two convolutional layers followed by three residual blocks, and a final linear layer. Here, we present some more for various measurement models. So far, in our analysis and experiments, we assumed that the parametric form of the measurement function and the distribution of those parameters is exactly known. This was then used for simulating the stochastic measurement process. Here, we consider the case where the parameter distribution is only approximately known. In this case, one would like the training process to be robust, i.e. the quality of the learned generator to be close to the case where the parameter distribution is exactly known. Through the following experiment, we empirically demonstrate that the AmbientGAN approach is robust to systematic mismatches in the parameter distribution of the measurement function. Consider the Block-Pixels measurement model (Section 4). We use the MNIST dataset. Pixels are blocked with probability p * = 0.5 to obtain a dataset of measurements. For several values of blocking probability p for the measurement function applied to the output of the generator, we train AmbientGAN models with this dataset. After training, we compute the inception score of the learned generators and plot it as a function of p in FIG0. We note that the plot peaks at p = p * = 0.5 and gradually drops on both sides. This suggests that our method is somewhat robust to parameter distribution mismatch. We provide further evidence that the generator learned through AmbientGAN approach captures the data distribution well. Generative models have been shown to improve sensing over sparsitybased approaches BID4 ]. We attempt to use the GAN learned using our procedure for compressed sensing. We trained an AmbientGAN with Block-Pixels measurement model (Section 4) on MNIST with p = 0.5. Using the learned generator, we followed the rest of the procedure in BID4 ] using their code 3. FIG0 (right) shows a plot of reconstruction error vs the number of measurements, comparing Lasso with AmbienGAN. Thus, we observe a similar reduction in the number of measurements while using AmbientGAN trained with corrupted samples instead of a regular GAN trained with fully observed samples.

represent the tldr for finding the corresponding scientific article

How to learn GANs from noisy, distorted, partial observations

Represent the TLDR for finding the corresponding scientific article

To understand the information stored in the latent space, we train a GAN-style decoder constrained to produce images that the VAE encoder will map to the same region of latent space.