Question: Represent the scientific article to find a short TLDR

Question: Variational Auto-Encoders (VAEs) are designed to capture compressible information about a dataset. As a consequence the information stored in the latent space is seldom sufficient to reconstruct a particular image. To help understand the type of information stored in the latent space we train a GAN-style decoder constrained to produce images that the VAE encoder will map to the same region of latent space. This allows us to''imagine'' the information captured in the latent space. We argue that this is necessary to make a VAE into a truly generative model. We use our GAN to visualise the latent space of a standard VAE and of a $\beta$-VAE. Variational auto-encoders (VAEs) have made a significant impact since their introduction by. However, one of their perceived problems is their reconstruction performance. This has spawned a wave of research into trying to improve the reconstruction performance (; ; ; ;). We argue that such attempts are misguided. The whole point of VAEs is to capture only compressible information and discard information specific to any particular image. This is a consequence of the well known evidence lower bound or ELBO objective function consisting of a negative log-probability of generating the original image from the latent representation (this is often implemented as a mean squared error between the image and the reconstruction, although as we argue in Appendix A this term should be proportional to the logarithm of the mean squared error) and a KL-divergence between the probability distribution representing a latent code and a'prior distribution' (usually taken as a multivariate normal with mean zero and unit variance). These two terms have a nice interpretation in terms of the minimum description length -this has been described elsewhere, for example,. The KL-term can be viewed as a measure of the amount of information in the latent code while the log-probability of the image measures the amount of information required to change the image produced by the decoder into the input image (see Section 3 for details). That is, the latent space of a VAE can be viewed as a model of the dataset-capturing compressible information while not encoding any image specific information (which is cheaper to communicate using the reconstruction loss). The great strength of a VAE is that it builds a model of the dataset that does not over-fit (i.e. code for in-compressible features found in specific images). However, because of this it typically will not do a good job of reconstructing images as the latent code does not contain enough information to do the reconstruction (for very restrictive dataset such as MNIST and Celeb-A a lot of information can be captured in the latent space, but for more complex datasets like ImageNet or CIFAR the reconstructions are poor). Of course, if you want good reconstructions on the training set then the simplest solution is to remove the KL-divergence term and just use an autoencoder. However, having a model that does not over-fit the dataset can be useful, but in this case the decoder of a standard VAE should not be regarded as a generative model-that is not its purpose. If we wish to generate realistic looking images we need to imagine the information discarded by the encoder. As a rather simplified analogy, consider a verbal description of an image "a five year old girl in a blue dress standing on a beach". If we asked different artists to depict such scene there is clearly not enough information to provide pixel-wise or feature-wise similarity between their interpretation although each artist could render a convincing image that satisfies the description. In a similar manner if we want a VAE to act as a generative model we need to build a renderer that will imagine an image consistent with the latent variable representation. A simple way to achieve this is using a modified Generative Adversarial Network (GAN). We call such a model a latent space renderer-GAN (or LSR-GAN). To generate an image we choose a latent vector z from the prior distribution for the VAE. This is passed to a generator network that generates an image,x, with the same dimensions as that of the dataset used to train the VAE. The generated image has both to convince a discriminator network that it is a real image-as is usual for a GAN -at the same time the VAE encoder should mapx close to z. To accomplish this we add an additional cost to the normal GAN loss function for the generator (L GEN) where q φ (·|x) is the probability distribution generated by the VAE encoder given an imagex and z is the latent vector that was put into the GAN generator. Note that when training the LSR-GAN we freeze the weights of the VAE encoder. The constant λ is an adjustable hyperparameter providing a trade-off between how realistic the image should look and how closely it captures the information in the latent space. This modification of the objective function can clearly be applied to any GAN or used with any VAE. Although the idea is simple, it provides a powerful method for visualising (imagining) the information stored in a latent space. Interestingly, it also appears to provide a powerful regularisation mechanism to stabilize the training for GANs. Combinations of VAEs and GANs are, of course, not new (; ; ; ;). In all cases we are aware of GANs have been combined with VAEs to "correct" for the poor reconstruction performance of the VAE (see Appendix B for a more detailed discussion of the literature on VAE-GAN hybrids). As we have argued (and expound on in more detail in Section 3), we believe that the decoder of a VAE does the job it is designed to do. They cannot reconstruct images accurately, because the latent space of a VAE loses information about the image, by design. All we can do is imagine the type of image that a point in the latent space represents. In the next section, we show examples of images generated by the LSR-GAN for both normal VAEs and β-VAEs (we also spend time describing VAEs, β-VAEs and the LSR-GAN in more detail). In addition, in this section we present a number of systematic experiments showing the performance of a VAE and LSR-GAN. In Section 3, we revisit the minimum description length formalism to explain why we believe a VAE is doomed to fail as a generative model. We conclude in Section 4. We cover more technical aspects in the appendices. In Appendix A we show that the correct loss function for a VAE requires minimising a term proportional to the logarithm of the mean squared error. In Appendix B we draw out the similarities and differences between our approach to hybridising VAEs with GANs and other work in this area. We present some additional experimental in Appendix C. A detailed description of the architecture of LSR-GAN is given in Appendix D. We end the paper with Appendix E by showing some samples generated by randomly drawing latent variables and feeding them to the LSR-GAN. A natural question to ask is what information about an image gets represented in the latent space of a VAE. To answer this we can use the VAE encoder to generate a distribution q φ (z|x) representing that image in the latent space (see Sections 2.1 for details on VAEs). From this distribution we can sample points in the latent space and feed this to the LSR-GAN generator. We show examples of this for both CIFAR-10 and ImageNet (down-sampled to 64 × 64) in Figure 1. In all cases in this paper the input images are taken from a test set that is independent of the training set. Note that both CIFAR-10 and ImageNet are "hard" for VAEs in the sense that they represent extremely diverse sets of images. As a consequence, the VAE latent space will struggle to store detailed information about the images and the VAE reconstructions will be poor. We have repeated this for a β-VAE (see section 2.3 for a full description of β-VAEs). We note that there is very little variation between the different samples drawn from q φ (z|x), particularly for the standard VAE (β = 1), showing that the latent space of the VAE is relatively smooth (there is more variation when β = 20). To get a sense of the variation in the information stored in latent spaces we show in Figure 2 inputoutput pairs, where the left image is the input and right image is the output generated by the LSR-GAN generator seeded with a latent vector encoding of the input image. The reconstructions capture the shape and , but clearly loses a lot of detail. In some cases it appears that the type of object is being captured, although in the case of the boat with the β-VAE (with β = 20) the wrong object is being rendered. The structure of a VAE is represented schematically below. We sample an input x from some dataset, D. To be concrete we will consider the case where the inputs are images, although clearly a VAE can be used to represent many different types of data. For each input x the encoder outputs a mean vector, µ, and standard deviation vector, σ, that describes an axis aligned normal distribution,. A latent variable z is sampled from this distribution and then fed to a decoder. For simple black and white datasets such as MNIST the decoder outputs a scalar at each pixel location that can be interpreted as the probability that the pixel is black. For more complex datasets the decoder ususal generates a "reconstruction"x = D θ (z). The probability of generating a pixel value x i is then usually taken as 2 )) and variance σ 2 that measures the expected size of the errors between the input images, x, and the reconstructions,x. The loss function for a VAE is equal to the negative evidence lower bound (ELBO) As explained in Appendix A, log(p θ (x|z)) is chosen to be proportional to the logarithm of the reconstruction error betweenx and the input image x-in our experiments this produced better reconstructions than replacing log(p θ (x|z)) with the mean squared error. LSR-GAN is a novel hybridization of VAE and GAN model. The most distinct difference of LSR-GAN from previous work is that it is a two-stage model. In the first stage we train the VAE model. Having done this we freeze the weights of the VAE and train the GAN. We train the discriminator, D, of LSR-GAN in the same way as a normal GAN. That is, we minimise a loss function where G is the generator or the decoder of LSR-GAN. The job of the discriminator, D is, to decide whether its import is a real image or not. Thus, to optimise the loss function we neet to maximize the log-probability of passing the real data, x, while minimising the log-probability of accepting a random sampling G(z) generated by a generator G seeded with a random latent vector z. The architecture of the generator is the same as that of a normal GAN but the loss function is slightly different. We add an additional term giving The parameters of the discriminator and generator are trained in the usual tick-tock fashion using gradient descent. We built the VAE and the generator of GAN using a ResNet as it gave slightly better performance than using a standard CNN. The architecture of the discriminator is the same as DCGAN . The architecture is described in Appendix D. To test the LSR-GAN we use the VAE to generate a latent representation z for an image drawn from an independent test set. The latent vector is then used as a seed value for the generator in the LSR-GAN. The LSR-GAN can get sharper reconstruction images than VAE (see Figure 3). Although not visually so obvious, we have used a quantitative measure of sharpness computed as luminancenormalised Laplacian (, Section 3.1.2). For the reconstructed images from the VAE we obtained a measure of 0.17 ± 0.03 while for the LSR-GAN we obtain 0.28 ± 0.08 (i.e. an improvement of a factor of two). We have also computed the FID measure of image quality for CIFAR-10 . For images seeded from a testing example the VAE achieved a score of 89.8 while LSR-GAN achieved a score of 44.1, while for images seeded with random latent variable (i.e. z ∼ N (0, I)) the FID score for the VAE is 138.6 while for the LSR-GAN it is 47.4. This should not be surprising. The decoder of the VAE is training only where there are training images. Despite the fact that the KL-divergence tries to ensure that as much latent space as possible is used, the constraint of minimising the reconstruction loss means that most of the latent space is far from a training example. Although the VAE does not do too badly generating testing examples, these tend to be substantially closer in the latent space to the training examples than random samples. In contrast the LSR-GAN is trained on random samples so that the generator will have to produce "realistic" images over the whole latent space. Of course, whether these generated images represents anything recognisable is open to question. For diverse training sets such as CIFAR-19 and ImageNet this may be very difficult. What image should we expect from a latent vector halfway between a truck and a bird? In Appendix E we show images generated by seeding LSR-GAN with random latent variables for CIFAR-10, ImageNet, MNIST and Celeb-A. A goal of generating a latent representation is for the representation to be disentangled. Intuitively disentanglement seems clear: We would want information that is somehow causally independent to be encoded into orthogonal directions (or different variables) in our latent space . Unfortunately, this is not only quite difficult to achieve in practice (at least, in an unsupervised setting), but it is even difficult to formulate (see). Despite this difficulty, there have been many attempts to achieve disentanglement (; ;). One of the most prominent has been the β-VAE introduced by , where the KL-divergence term in a normal VAE is weighted by a parameter β The argument is that by making β 1 we encourage disentanglement. Contrariwise, by making β 1 we make a VAE closer to an auto-encoder. This improves the reconstruction performance on the training examples, but at the cost of allowing the latent space to over-fit the training set. In Figure 4 we show examples of input-output pairs for different values of β. We observe that for large β the outputs are quite different from the input images in contrast to small β where many more details of the original input are captured. Although the LSR-GAN model generates slightly clearer, less blurry, images, it has a lower reconstruction error than the VAE decoder. We show the mean squared error measured on a testing set from CIFAR-10 as a function of β in Figure 5 (a). This poor performance of the LSR-GAN is unsurprising, it uses the same information as the VAE (i.e. the information stored in the latent space). By producing sharper images it will pay the price of getting the boundary wrong. The blurry edges from the VAE is a way to hedge its bet and reduced the mean squared error. Interestingly, the mean squared error remains fairly constant as we increase β from a low value, until we reach β = 1 after which it rapidly increases. One interpretation of this fact is that the VAE with β = 1 is successfully encoding all the useful information (i.e. compressible information) so for reconstructing unseen images it will perform as well as an auto-encoder. As we increase β above 1, the reconstruction error increases rapidly. In Figure 5 (b) we show the classification performance as measured by a simple classifier trained on the CIFAR-10 training set. The classifier performance achieved an 84% correct classification on the raw images. We find little variation as we decrease β below 1. As we increase β above 1 the classification accuracy falls off. Again we can attribute this to the latent space of the VAE (with β = 1) capturing most useful information. Interestingly the high-β VAE fails to capture (a) (b) Figure 5: Performance of the VAE (blue points) and LSR-GAN (red points) versus β. In (a) we show the means squared error, while in (b) we show the classification performance using a classifier taking the reconstructed images. The images are taken from CIFAR-10. Error bars show ±1 standard error. "objectness" well. This suggests that, at least for CIFAR-10, the type of object does not contain very much information about its appearance and is rapidly discarded. To understand what VAEs do it is useful to interpret them in the framework of the minimum description length (MDL) formalism. In MDL we consider communication a dataset D through a communication channel using as few bits as possible. We can do this using lossy compression, where we encode each input x by a code z, which we communicate down our channel. The receiver decodes the message and produces an approximation of the inputx. To communicate the original information we send the code z together with the error = x−x between the input x and the reconstructionx. Because the distribution of errors, p, is more sharply concentrated than the original inputs, p(x), this method allows us to communicate the image more efficiently than transmitting the raw pixel values. The expected cost of transmitting an input is where M (z) is the number of bits needed to communicate the code, z, and E is the number of bits required to communicate the error,. In the MDL formalism we attempt to find a code that minimises the description length L. To communicate the model and errors we need to use an optimal coding strategy. Rather than specifier and actual code we can use the Shannon bound (i.e. the negative log-probability of the tokens we transmit). For this to be meaningful, we need to specify both the errors and code to a finite precision. The precision of the errors will determine the accuracy of the data we communicate. If the i th component of the error is distributed according to p(i) then the cost of communicating the error to a precision of ∆ is approximately − log(p( i) ∆) = − log(p( i)) − log(∆). The factor − log(∆) is common to all coding schemes so is irrelevant to choosing optimal codes z. In contrast the precision to which we transmit the model will directly determine the cost M (z). There is a balance to be struck: a more precise model can potential lead to a better reconstructionx, reducing the reconstruction cost, E, but at the same time increasing the cost, M (z), of communicating the code z. The KL-divergence term, KL q(z) p(z) (also known as the relative entropy) can be interrupted as the communication cost (in nats) of transmitting a random variable z with uncertainty given by q(z) assuming an underlying probability distribution of all random variables of p(z). Using this interpretation we see that the loss function of a VAE is equivalent to the expected message length (in nats) of communicating a sample from the dataset D by using a random variable z with uncertainty q(z). By minimising the loss function we find a coding scheme with the minimum description length (or, at least, an approximate local minimum). By encoding a message as a random variable z drawn from a distribution q φ (z|x) the VAE is able to find an optimal balance between accuracy to which it transmits the model (determined by the standard deviation vector, σ, generated by the VAE encoder) and the need to reduce the reconstruction error. From an MDL perspective the ELBO is the correct objective function, and should not be regarded as a approximate lower bound to what we really want to achieve. If there are too many dimensions in the latent space then some of the components of z (channel in information theory terms) are such that z i is approximated distributed by N (z i |0, 1) for all inputs x. The channel is effectively "switched off" (and it will be ignored by the decoder as it is just a source of random noise). This is referred to as latent variable collapse and is sometimes viewed as problematic, however, from the MDL viewpoint it acts as an elegant automatic dimensionality selection technique. The job of the decoder in a variational autoencoder is to reconstruct the image only using information that can be compressed. Image specific information is ignored. For example, information about the precise shape of an object is probably not compressible. As a the decoder tends to hedge its bets and has a blurry outline. Of course, some encoders and decoders will be better than others, but to date there is little evidence in the literature that the performances of VAEs are massively suboptimal, at least, when working with images. With an extremely powerful encoder and decoder and a limited dataset it would be possible for the encoder to communicate an identifier of the input image and for the decoder to reproduce the image just from the identifier, thus avoiding communicating any information about the visual content of the image-this requires that the decoder memorises all the images. This would be an extreme case of what is sometimes called posterior collapse. There is some evidence that with very strong encoders and decoders that the amount of information stored in the latent space (as measured by the KL-divergence) decreases . This might point to a weakness of the VAE set-up-the MDL set-up really only makes sense when the dataset is arbitrarily large-, but this problem could be ameliorated by data augmentation. However, using standard CNN encoders and decoders we found no evidence for memorisation of the images (for example, the VAE would produce a similar level of reconstruction for images from a separate test set). For language modelling there seems to be more evidence that VAEs often fail to extract information in the latent space, but for images it seems likely that a properly trained VAE will extract a good fraction of the compressible information. We believe that the failure of the VAE decoder to produce high quality reconstructions (except in the case very of simple datasets such as MNIST and possibly CELEB-A) is because to do so would require communicating information that is noncompressible. As a consequence we should not think of the decoder of a VAE as a generative model: It will, by design, produce blurry and poor quality reconstructions. We want this to ensure that the latent space only captures information that is common across many images. We see the mapping from images to latent space as a many-to-one mapping. Thus, the mapping from the latent space to images will be ambiguous and the best we can do is imagine an image compatible with the latent variable: exactly what we have designed the LSR-GAN to do. VAEs are often taken to be a pauper's GAN. That is, a method for generating samples that is easier to train than a GAN, but gives slightly worse . If this is the only objective then it is clearly legitimate to modify the VAE in anyway that will improve its performance. However, we believe that this risks losing one of their most desirable properties, namely their ability to learn features of the whole dataset while avoiding encoding information specific to particular images. We have argued that because of this property, a VAE is not an ideal generative model. It will not be able to reconstruct data accurately and consequently will struggle even more with generating new samples. One of the weaknesses of the vast literature on VAEs is that it often attempts to improve them without regard to what makes VAEs special. As we have argued in this paper, a consistent way of using the latent space of a VAE is to use a GAN as a data renderer, using the VAE encoder to ensure that the GAN is generating images that represent the information encoded in the VAE's latent space. This involves "imagining" the information that the VAE disregards. LSR-GAN can be particularly useful in generating random samples, although, as shown in Appendix E, for very diverse datasets the samples are often not recognisable as real world objects. Although there are already many VAE-GAN hybrids, to the best of our knowledge, they are all designed to "fix" the VAE. In our view VAEs are not broken and "fixing" them is actually likely to break them (i.e. by encoding image specific information in the latent space). Although, the main idea in this paper is relatively simple, we believe its main contribution is as a corrective to the swath of literature on VAEs that, in our view, often throws the baby out with the bath water in an attempt to fix VAEs despite the fact that perform in exactly the way they were designed to. A ON THE ELBO In the standard VAE we maximise the log-probability of generating the original image. In the original paper this was achieved by the decoder outputting a probability distribution akin to what happens in the latent space. More often it is assumed that the pixel errors are normally distributed with some variance σ 2. Thus the log-probability of generating all the images is where the sum is over all predicted pixels-i.e. the number of pixels in an image times the number of colour channels times the number of examples (or, more usually, the mini-batch size). However, (at least, if we make the natural assumption that the errors have mean zero). As a consequence 2 so that we should minimise N log σ 2 /2. In information theory terms this tells us that it cheaper to communicate the residues if they are more tightly concentrated. Note that since σ 2 is proportional to the mean squared error, E MSE, it suffices to minimise N log(E MSE) /2. We note that which is precisely the gradient of if we ignored the dependence of σ 2 onx i. In many publically available implementations of VAEs the algorithm minimises 2 which arbitrarily assumes σ 2 = 1 2 rather than its true value. This means that these implementations are effectively running a β-VAE with some unknown β (in our experience with β > 1). This makes comparing from different VAE implementations difficult. For example, rescaling outputs to lie in the range [−1, 1] rather than would change the effective β-value. The hybridisation of VAE (or autoencoder) and GAN models have been developed for several years. There are many attempts on this area and we compare LSR-GAN to the most related work in this section. The adversarial types autoencoder is the most intuitive and simplest way to combine a VAE or an autoencdoer and a GAN models. Most of these models introduce a discriminator into the autoencoder training. AAE applies a discriminator to distinguish the output of encoder and the random sample from the prior distribution. It uses this discriminator to replace the KL term in VAE. VAE/GAN is the first model that applied feature-wise errors and the input of its generator contains three different types images: the reconstruction images, the generated images and the real images. The same as our model, it collapse the decoder and the generator into one. MDGAN is another AE-GAN hybrid which is close to VAE/GAN, they try to match the manifold of GAN to real data by adding a geometric metrics regulariser and mode regulariser. None of these methods feed the output of generator back into the encoder or train their network in two-stages, which is the biggest difference between these methods and ours. Also, many of these hybrid models adopt an autoencoder instead of VAE while the VAE in our model cannot be replaced by an autoencoder. There are not many models that use the output of decoder to feed the encoder. The Introspective Adversarial Network (IAN) is a unified model which means the discriminator is not separate. IAN only encodes the feature that extracted by discriminator rather than the raw images. The discriminator of IAN extracts features from both raw images and synthetic images. The generator accept both random sample and the output of the discriminator as inputs at the same time. In contrast, our models only accept one input. Another model that adopts the introspective method is IntroVAE , it constructs the inference model E and generator model G in a circulation loop. IntroVAE has the ability to generate high-resolution images. But it does not contain any discriminator network. The most closely work to our LSR-GAN is VEEGAN . It introduces a second network F θ to the GAN. The task of F θ is to map both the real images and synthetic images to a Gaussian distribution which is what we ask the encoder to do. When the input of F θ is the output of generator, the objective function minimise the distance between the input of generator and the output of F θ. If the input of F θ is real data, the objective function minimise the cross entropy between Gaussian prior and the output of F θ. Another related model is the Generative moment matching networks (GMMN) . In this model the autoencoder is frozen and they then minimize the maximum mean discrepancy (MMD) between the generated representation and data representation, and they use an uniform prior to generate the representations. In LSR-GAN, we match two Gaussian distributions in maximizing the probability distance. None of these related works are two-stages models except GMMN. Also, to the best of our knowledge, LSR-GAN is the first VAE-GAN hybrid model that applies the probability distance in the loss function. We briefly present some additional experimental data. In Table 1 we present measurements of the performance of outputs from both VAEs and LSR-GAN for different values of β. Some of this data is also presented graphically in Figure 5, but we have included additional measurements. The performance of the LSR-GAN depends on the hyper-parameter λ. This balances the need to produce convincing images (from the discriminator's point of view) with the requirement that the latent space of the GAN should be close to that for the VAE. These two objectives are not necessarily contradictory, although we will see that changing λ has benefits and drawbacks. In Figure 6 we show the effect of changing λ over approximately three orders of magnitude on (a) the absolute classification accuracy (b) the classification accuracy compared to the class labels predicted by the classifier on the raw images (c) the mean squared reconstruction error and (d) the variance in the predictions when choosing different samples from q φ (z|x). We see that increasing λ improves the classification performance (both relative and absolute). However, and perhaps surprisingly, increasing λ produces a significant reduction in the reconstruction error. More intuitively it also causes a reduction in the variance between images sampled independently from q φ (z|x). That is, using the encoder in the LSR-GAN acts a regulariser ensuring close by points in latent space map to similar images. More details are given in Table 2. (a) (b) (c) (d) Figure 6: Graphs showing the classification performance of images generated by our GAN with different λ values (on test dataset). The left-bottom graph shows the mean square error between reconstruction images and imagination images. The right-bottom graph shows the variance between images generated by our GAN from latent representations of an input image. The x-axis is the log value of different λ, the errors are too small which make error bars look like lines. In this appendix we describe the detailed architecture of the VAE and LSR-GAN we used. Table 3 describes the structure of the VAE's encoder and decoder and the GAN's generator and discriminator networks. The encoder and decoder/generator are based on a ResNet. The ResNet block structure is shown in Figure 7. Both networks are optimized using Adam with a learning rate of 2 × 10 −4 and β 1 = 0.5. The code we used to implement the models is available at https://github.com/iclr-2020-zzz/LSR-GAN. In this appendix we show sample images generated by LSR-GAN starting with a random seed z ∼ N (0, I). These are shown in Figure 9 for an LSR-GAN trained on CIFAR-10 and ImageNet. Although the images superficially look reasonable on close inspection it is clear that most samples for the LSR-GAN trained on CIFAR-10 and ImageNet are not real world objects. This reflects the fact that the images for these two dataset are very variable leaving most of the latent space representing rather surreal objects. Figure 8: Random samples generated by LSR-GAN trained on CIFAR10 and ImageNet with β =1 and λ=1. We have also trained LSR-GAN on MNIST and Celeb-A with samples shown in Figure 9. Perhaps unsurprisingly, most samples are identifiable. Figure 9: Random samples generated by LSR-GAN trained on MNIST and Celeb-A with β =1 and λ=1.

To understand the information stored in the latent space, we train a GAN-style decoder constrained to produce images that the VAE encoder will map to the same region of latent space.

Represent the TLDR for finding the corresponding scientific article

In this paper we introduced an algorithm to learn the connectivity of deep multi-branch networks. The approach is evaluated on image categorization where it consistently yields accuracy gains over state-of-the-art models that use fixed connectivity.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Human brain function as measured by functional magnetic resonance imaging (fMRI), exhibits a rich diversity. In response, understanding the individual variability of brain function and its association with behavior has become one of the major concerns in modern cognitive neuroscience. Our work is motivated by the view that generative models provide a useful tool for understanding this variability. To this end, this manuscript presents two novel generative models trained on real neuroimaging data which synthesize task-dependent functional brain images. Brain images are high dimensional tensors which exhibit structured spatial correlations. Thus, both models are 3D conditional Generative Adversarial networks (GANs) which apply Convolutional Neural Networks (CNNs) to learn an abstraction of brain image representations. Our show that the generated brain images are diverse, yet task dependent. In addition to qualitative evaluation, we utilize the generated synthetic brain volumes as additional training data to improve downstream fMRI classifiers (also known as decoding, or brain reading). Our approach achieves significant improvements for a variety of datasets, classifi- cation tasks and evaluation scores. Our classification provide a quantitative evaluation of the quality of the generated images, and also serve as an additional contribution of this manuscript. Functional Magnetic Resonance Imaging (fMRI) is a common tool used by cognitive neuroscientists to investigate the properties of brain function in response to stimuli. Classic analysis approaches BID19 focused on analyzing group-averaged brain function images. However, it was discovered that brain activation patterns vary significantly between individuals. Thus, modern analysis now prioritizes understanding the inter-subject variability of brain function (; BID6 . Our work is motivated by the view that generative models provide a useful tool for understanding this variability -as they enable the synthesis of a variety of plausible brain images representing different hypothesized individuals, and high-quality generative models can be analyzed to posit potential mechanisms that explain this variability BID10 . The presented in this paper provide -to our knowledge for the first time, positive suggesting that it is indeed possible to generate high quality diverse and task dependent brain images. While we can qualitatively evaluate generative brain images, quantitative evaluation allows us to objectively compare between various . To this end, we utilize the generated synthetic brain volumes as additional training data to improve downstream fMRI classifiers. The use of classifiers to predict behavior associated with brain images is also known as decoding or brain reading BID21 BID16). Classifiers such as support vector machines and deep networks have been applied for decoding brain images. For example, attempted to classify which of 10 categories of object a subject was looking at (including similar categories, such as horses and cows) based on limited number of brain images. Besides visual tasks, BID14 distinguished active regions of brains when subjects listened to linguistic words, where the stimuli included five items from each of 12 semantic categories (animals, body parts etc.).Beyond providing a model for individual variability, high quality brain image synthesis addresses pressing data issues in cognitive neuroscience. Progress in the computational neurosciences is stifled by the difficulty of obtaining brain data either because of a limited culture of data sharing, or due to medical privacy regulations BID18. For the computational neuroscientist, generated images deliver unlimited quantities of high quality brain imaging data that can be used to develop state of the art tools before application to real subjects and/or patients BID21. This approach of using modern generative models to synthesize data, which in turn accelerates scientific study, has already proven useful in many scientific fields such as particle physics and astronomy BID1. Our work represent a first application for this approach to neuroscience. One of the promising generative models are Generative Adversarial Networks (GANs) BID7, capturing complex distributions using a non-cooperative two-player game formulation: a generator produces synthetic data by transforming samples drawn from a simple distribution; a discriminator focuses on distinguishing synthetic and real data. Despite (or possibly due to) the compelling formulation, GAN training is known to be unstable. To address this difficulty various variants have been proposed. Wasserstein GANs (WGANs) formulate the objective using the Wasserstein distance rather than the classical Jenson-Shannon divergence. Improved training of WGANs BID9 applies an additional gradient penalty, which avoids the critical weight clipping in WGANs which might lead to pathological behavior. Dualing GANs restrict the discriminator and formulate the dual objective . Beyond improving the stability, conditional GANs make it possible to control the data generation process by conditioning the model on additional information. Auxiliary Classifier GANs (AC-GANs) BID15 ) unify a GAN and classifier to a single architecture, employing labels to generate ImageNet samples. 3D GANs BID23 reconstruct 3D objects and BID4 propose to use improved WGAN to enhance the stability of 3D GANs. We make the following contributions in this paper:1. We develop Improved Conditional Wasserstein GANs (ICW-GAN) and Auxiliary Classifier and Discriminator GANs (ACD-GAN), two types of 3D conditional GANs to synthesize fMRI brain data, both of which we find to be stable to train. 2. We assess the qualitative quality and diversity of generated brain volumes. Our suggest that the proposed models are able to generate high quality task-dependent and diverse 3D brain images. 3. We evaluate our models on three datasets using a series of image classification tasks with support vector machines and deep network classifiers at two levels of brain image resolution. Results show that augmenting training data using synthetic data generated by our models can greatly improve test classification accuracy of brain volumes. We are unaware of any published papers using neural networks to generate brain imaging data. However, neural networks have been used by several authors for classifying brain imaging data. BID5 and BID12, used 2D deep nets to extract features of fMRI brain images to classify brain states. BID14 applied both 2D and 3D neural networks to classify fMRI brain data. BID20 decoded fMRI data of video stimuli and classify data into visual categories. Similarly, BID14 extracted features from 4-D fMRI data and used deep learning methods for discrimination of cognitive processes. To learn a distribution over data x, a GAN BID7 formulates a 2-player noncooperative game between two deep nets. The generator G uses a random noise vector z sampled from a prior distribution P z (z) as input and produces an image G(z). The generator is trained to fool the discriminator, which receives either synthetic data or real data and differentiates between them. Formally, G and D play the following two-player minimax game with value function V (G, D): DISPLAYFORM0 For the conditional GAN , both G and D are conditioned on some extra information y, for instance, class labels or other features. The conditioning can be presented by feeding y into both the generator and the discriminator. This work is successful in one-to-many mappings such as image labeling with many tags. The generator of our ICW-GAN. The 128 dimensional encoding z is drawn from a multivariate normal distribution. The label vector is a one-hot encoding, i.e., the one entry that equals one represents the class that the generated volume belongs to. It is concatenated to input and hidden layers and for each of these layers, a fully connected layer transforms the label vector to a volume. Our stride in the de-convolutional layers is in the batch, height, width, length, feature map dimension. The Wasserstein GAN (WGAN) uses Wasserstein-1 distance W (q, p) as the objective function. It computes the minimum cost of transporting mass to transform the distribution q into the distribution p. Under this circumstance, W (q, p) is continuous everywhere and its gradient with respect to its input was found to be more stable than its classical GAN counterpart. Improved Wasserstein GAN (IWGAN) BID9 argues that weight clipping of the critic, i.e., the discriminator, in WGANs inevitably causes the gradient to either vanish or to explode. To address this issue BID9 propose an alternative penalty term in the critic loss based on the gradient norm. In an auxiliary classifier GAN (AC-GAN) BID15, every generated sample has a corresponding label which can be used as a condition in the generator and the classifier. The discriminator is modified to contain an auxiliary decoder network to reconstruct the training data class. Further, 3D-GANs BID23 extend GANs to 3D object generation. Different from classical GANs, 3D-GANs apply the three dimensional convolution in both G and the D. By learning deep object representations, 3D GANs can generate visually appealing yet variable 3D object volumes. In the following we introduce two models for fMRI data generation: ICW-GAN and ACD-GAN. They differ in model structure and in the way label information is taken into account. We first show the model structure of 3D ICW-GAN and its downstream classifiers in Section 3.1, and present ACD-GAN in Section 3.2. Similar to classical generative adversarial networks (GANs), ICW-GANs are formulated as a noncooperative two-player game between two adversaries: a generatorx = G θ (z), which generates artificial samplesx from randomly drawn latent encodings z via a transformation using a deep net parameterized by θ; and a discriminator D w (x) represented via the logit obtained from a deep net parameterized by w. Following Arjovsky et al. FORMULA0, we build our models based on minimizing the Wasserstein distance via min DISPLAYFORM0 where {D w} w∈W denotes a set of functions that are K-Lipschitz for some K. IW-GANs BID9 ) provide a more stable method for enforcing the Lipschitz constraint by employing a gradient penalty rather than weight clipping, when optimizing: DISPLAYFORM1 Herebyx is a convex combination of real data and artifical samples, i.e.,x ← x + (1 −)G(z) with drawn from a uniform distribution (∼ U).In our models, we extend IW-GAN in two directions. First, because fMRI data is three dimensional, 3D convolution and deconvolution are used in order to maintain the spatial structure and voxel information of the fMRI data. Second, we condition both the discriminator D w and the generator G θ on available labeled data. As shown in FIG0, our generator consists of three fully convolutional layers. In convolutional layers, we use kernels with a size 4 × 4 × 4 and strides 2, with batch normalization and Leaky ReLU layers added between and a tanh layer at the end. The discriminator architecture is a mirrored generator, but no activation function is used at its end. This ICW-GAN model follows models proposed in BID9; BID23, however, we found the following modifications to stabilize training and supervise the generation process for classification data: compared to existing GANs, our ICW-GAN applies a more stable upper bound for Wasserstein distance as an objective function, leverages conditioning labels to supervise the generation process, and utilizes 3D convolution methods for 3D object generation. To include label information, we concatenate labels to the input and to the hidden layers. At the input of the generator, one-hot labels are combined with the brain vector. Then, for each of the intermediate layers, we use a fully connected layer followed by a tanh activation to transform the one-hot vector to a volume of appropriate size, i.e., 15 × 2 × 2 for the first hidden layer, and 15 × 3 × 3 for the next. We empirically found dimensions other than 15, e.g., 3 to work well too. We concatenate the label volume to intermediate volumes on feature dimension and pass the joint one to the next deconvolution layer. We follow the same procedure in the architecture of the discriminator. Referring to the volume labels via y, the objective function L of our ICW-GAN model is as follows: DISPLAYFORM2 wherex DISPLAYFORM3 λ is a gradient penalty coefficient. We optimize both the discriminator and generator loss using Adam optimizer BID11.DOWNSTREAM CLASSIFIERS FOR ICW-GAN Our work is partially motivated by the difficulty of acquiring fMRI data, so we consider the task of enlarging the dataset for downstream tasks such as classification, using synthetically generated samples. To assess the suitability of the developed generation procedure, we employ a variety of downstream classification processes. More specifically, we train a classifier with real ('Real') or real plus generated ('Real+G') data. We also develop a conditional GAN architecture which jointly trains the generator, the discriminator and the classifier. This form of training is inspired by the AC-GAN BID15 formulation. Our generator architecture is illustrated in FIG0 and identical to the one employed for the ICW-GAN. The combination of classifier and discriminator is provided in FIG1. As before, the discriminator assesses the logit of the probability that a provided sample is real or artificially generated. In addition, the classifier provides a probability distribution over the class labels. In this model, training passes labels through the generator and the discriminator in order to condition the adversarial process. However, no labels are employed in the classification stream. The classifier loss, L C, is the log-likelihood of the correct class. Compared to L in Eq., the discriminator loss, L D includes the additional classifier component DISPLAYFORM0 The generator loss function, L G, consists of the likelihood of fake images and one part of L C, i.e., DISPLAYFORM1 Similarly, the discriminator loss function is given by L D = L + L C. We optimize the generator, classifier and discriminator loss functions using Adam optimizer BID11. We examine the performance of the two aforementioned models on three different datasets from Neurovault BID8 of generated fMRI data for classifier training, we focus on two variants of the model architecture for generating images. The original images are downsampled using the nilearn python package 2 to 13 × 15 × 11 (lower resolution) and 26 × 31 × 23 (higher resolution) spatial resolutions for collection 1952, and 13 × 15 × 11 (lower resolution) and 26 × 31 × 22 (higher resolution) resolutions for the other two collections. In Section 3, we present the architectures of our models for generating low resolution data, which only have 3 convolutional layers. For high resolution data we add one more convolutional and deconvolutional layer in the discriminator and generator respectively. For high resolution data, the classifier also uses 4 convolutional layers in the ACD-GAN. We first show qualitative for generated 3D volumes, then we provide quantitative for 3D volume classification. We use cross validation in all reported to carefully assess the classifier performance. See the supplementary material for details. Dataset 1: Collection 1952 Collection 1952 has 6573 brain images, 45 classes with a total number of 19 sub-classes. The labels include task description that subjects were going through during the imaging process, such as visual, language and calculate. A combination of 2 to 4 sub-classes are subsumed for a class. There are a total of 45 classes because not every potential combination is observed. In our experiments, we work on a subset of 36 classes because 9 of the 45 classes have less than 30 examples. If a class has more than 100 images, we split images in that class 7:1:2 into training, validation and test datasets. Below 100 images but more than 30 images we use a 3:1:2 data split. Note that we ensure that the test data always consists of real images only. We train the ICW-GAN and ACD-GAN for 2500 epochs with a learning rate of 1e-4 and 50% exponential decay for each 3000 iterations and a batch size of 50. 2-D projections of several brain volumes generated by ICW-GAN are illustrated in FIG2. The projections in the top row of FIG2 class'visual words, language, visual,' and those at the bottom are from class'places, visual, right hand.' By examining the generated brain images, we find diversity in both intra-class and betweenclass. We refer the interested reader to the supplementary material for a comparison of generated and real brain images. Inspired by BID15, we compute the multi-scale structural similarity (MS-SSIM) BID22 to examine the intra-class diversity of the data generated by ICW-GAN. The MS-SSIM score is a similarity metric based on image similarity at different resolutions. Its values are within. Higher MS-SSIM values correspond to more similar images. We measure the mean MS-SSIM score with 100 randomly chosen volumes of mixed data with a given class, and ones of real training data. In 13 × 15 × 11 resolution, we find that 22 classes with mixed data have a lower MS-SSIM score than only with real training data. In other words, 61.1% classes with mixed data have sample variability that exceeds those only with real training data. In 26 × 31 × 23 resolution, 69.4% classes with mixed data have a lower MS-SSIM score. See Figure 4 for details. To further assess the quality of the generated data we evaluate classification performance. Note that the test data is always composed of real images. The classification are shown in Table 1. Downsampling refers to the downscaling factor as already discussed. The second column indicates the type of training data we use for the classifier: only using real data ('Real'), or using the mixed data of real and generated volumes ('Real+G'). The third column denotes the classifier type, i.e., an SVM or a deep net ('NN'). We use the validation dataset to choose the best training models and use these models to classify the test data. We observe the deep net classifier to generally outperform SVMs and we observe generated data to be beneficial for classifier training. Dataset 2: Collection 2138 For data in collection 2138, there are 1847 brain images, 61 classes and 50 labels. Because of the small size of the dataset, we randomly choose 70% of the brain images as training data and leave 30% as test data. In this case, we do not have development data to supervise the training process; thus, we train our models for 1000 epochs in several runs and record the best classification , which are summarized in TAB3. In this collection, we downsample brains by a factor of 8 and 4 to volume sizes of 13 × 15 × 11 and 26 × 31 × 22 respectively. Similar to the Accuracy Macro F1 Precision Recall Table 4: Results of ACD-GAN on the three datasets in the low resolution setting. We find the ACD-GAN to perform slightly better than the ICW-GAN.earlier reported we observe deep nets to outperform SVMs, but more importantly, generated data during training was again suitable to improve the classifier performance on real data. Dataset 3: Collection 503 5067 brain images are in collection 503. Subjects in this dataset are required to respond to 30 images from the International Affective Picture Set BID2. These 30 images were used to train the Picture Induced Negative Emotion Signature also described in this paper. We consider the 30 pictures classes of brain data. In other words, there are overall 30 classes and each class represents a picture shown to subjects. This collection needs extra preprocessing because brain volumes are available with two shapes: 79 × 95 × 68 and 91 × 109 × 91.We conform large volumes to the small ones. Similarly, all experiments are at the two levels of resolution. Classification are summarized in TAB5 and again follow the trend reported earlier. Results of Using ACD-GAN Since our ICW-GAN achieves the best performance in the three databases when we use mixed training data, we compare these best to of ACD-GAN at the low resolution. 2-D projections of brain volumes generated by ACD-GAN are shown in FIG3. In Table 4, we present classification of our models and show that ACD-GAN obtains improved accuracy scores compared to a ICW-GAN for most of the investigated metrics. As a baseline, we also implemented the recent generative model -the 3-D. According to all the presented above, the proposed methods achieve much improved overall better performance on the three datasets. This is true at both levels of resolution for brain images. The also outperform a 3-D AC-GAN applied to the same dataset. We also observe that our strategy of mixing real and generated data does not always yield the best . For example, using real data only to train the NN classifier achieves the largest macro F1 score at high resolution in TAB5. However, the proposed approach consistently improves accuracy metrics -which is by far the most popular metric for evaluating multi-class classification performance. Generative models provide a useful tool for understanding the individual variability of brain images. The of this manuscript show -to our knowledge for the first time, that 3-D conditional GANs, in particular our proposed ICW-GAN and ACD-GAN, can generate high quality diverse and task dependent brain images. We hope our inspire additional research on generative models for brain imaging data. Beyond qualitative evaluation, we evaluate quantitative performance by using the generated images as additional training data in a predictive model -mixing synthetic and real data to train classifiers. The show that our synthetic data augmentation can significantly improve classification accuracy -a which may be of independent interest. Future work will focus on additional qualitative evaluation of the generated images by neuroscience experts and exploration of various applications. We also plan to more throughly investigate the trained models to further explore what it may contribute to the science of individual variability in neuroimaging. Finally, we plan to expand our models to combine data across multiple studies -each of which use different labels, by exploring techniques for merging labels based on the underlying cognitive processes BID17.6 SUPPLEMENTARY MATERIAL First, we explain our cross-validation strategy. In Figure 6, the blue bar represents training data and the green bar is test data. For 3-fold cross-validation, first we partition test data to three folds and each fold maintains a similar proportion of data for each class as original test data. For each round of cross-validation, we leverage one of the three parts as test data and left two folds together with training data to train a model. The same strategy is used with generated data to ensure that test performance is only based on the test data. Each evaluation metric is the average of the three rounds. Figure 6: Cartoon illustration of our 3-fold cross validation strategy. We tested our model with various cross-validation settings and calculated the variances of the evaluation metrics (Table 5). We conducted this experiment on low resolution data from collection 1952 and synthetic data generated by the ICW-GAN. Except for the number of folds used when partitioning the test dataset, the training strategy for 5-fold and 10-fold cross-validation is similar to that of 3-fold cross-validation. The small variances suggest that the reported accuracy differences, while small, are indeed significant. Table 5: Accuracy, F1, Precision, Recall and their variance (column 3,5,7,9) for 3-fold, 5-fold and 10-fold cross validation. We conducted this experiment with the training data of mixed'Real+Synth.' data of collection 1952 in the low resolution (4.0×) setup. We present the ICW-GAN training loss curve evaluated on the validation dataset of collection 1952 in FIG6. This illustrates the stability of the proposed improved Wasserstein variants. In order to further evaluate the quality of the generated data for collection 1952, we trained using only generated data and tested with real data. We used the deep net classifier for evaluation in this case while varying the number of input samples for each class. FIG5 shows that in general, the test accuracy improves as the amount of artificial training data increases. The red and yellow lines show the accuracy obtained when only using real data. These suggest that the generated data can be as effective for classification as real data with sufficient numbers of samples. Table 6: Results of synthesizing data using collection 1952 (low resolution). Comparison between the GMM and ICW-GAN. We list 6 training data strategies: in the 1st, 2nd, 4th and 5th row, we only use synthetic data to train the deep net classifier while in the 3rd and 6th row, we mix real and synthetic data together to train the same classifier. We also used a Gaussian Mixture Model (GMM), i.e., trained separately for each class, as a simple baseline generative model. The are shown in Table 6, where 6 training strategies were listed and the deep net classifier (same as the NN classifier) was used for all the strategies. I.e., we leveraged 20 synthetic images in each class (20×number of classes in all) to train the NN classifier. Note that both generative models learned the probability distribution from collection 1952. The evaluation scores of our models are significantly higher than the ones obtained with a GMM, particularly when using only synthetic data to train the classifier.6.5 GENERATED IMAGE QUALITATIVE ANALYSIS We used NeuroSynth 3, a brain decoding tool to qualitatively analyze generated volumes. Its principle is to compare activity regions of a given brain volume to a data pool and then to obtain an analysis report about correlation to specific activity patterns. Figure 9 shows a generated brain image of collection 1952 by the ICW-GAN in low resolution and its analysis report with the top 10 ranking list of correlation. The supervised class of this generated brain is "Non-human sound, auditory", and several correlations with top rank are related to auditory and sound. Figure 9: The projections of a generated brain volume (left) and its corresponding analysis report (right). The volume belongs to the class'Non-human sound, auditory.' We also examine our models using a multilabel classification strategy in low resolution data of collection 1952. As described before, one class in collection 1952 consists of several labels and the collection has a total of 19 sub-labels. Instead of encoding the classes with a one-hot vector of 45 dimensions, we encode them in 19 dimensions, each of which represents a sub-label. The value in a dimension is 1 only if the sample possesses that label, otherwise, 0, i.e., a sample comes with a 19 dimensional binary vector. This representation is demanding since the probability space is significantly larger, specifically, 2 19 possibilities, for a classifier to learn and discriminate. We use SVM to train real and mixed training data (synthetic data obtained from ICW-GAN), both of which Table 7: Multilabel for collection 1952 at 4.0× downsampling and synthetic data were from the ICW-GAN. Even in this highly demanding task, accuracy scores with mixed data (the second row) outperform the baseline (the first row).are in the resolution of 13 × 15 × 11, and utilize 3-fold cross-validation as discussed in Section 6.1. Several evaluation scores are shown in Table 7. We observe that even in this highly demanding task, accuracy scores with mixed data (the second row) outperform the baseline (the first row). We present several real images of collection 1952 with their labels above the 2D projections of brain images in FIG0.Figure 10: Real images in collection 1952 at 4.0× downsampling with labels on top. Linear normalization is used and the lower threshold for plotting these images is 0.48. More synthetic brain images using ICW-GAN are shown in this section. FIG0 show generated ones of low resolution from collection 1952, 2138 and 503 respectively. For collection 1952, classes are given above the images. As for collection 2138, the corresponding categories of the classes are as follows:• 14: visual form recognition, feature comparison, response selection, response execution, relational comparison, visual pattern recognition • 22: response execution, working memory, body maintenance, visual body recognition • 4: response selection, response execution, punishment processing • 43: motion detection • 2: response selection, response execution, animacy perception, animacy decision, motion detection • 3: response selection, response execution, motion detection

Represent the short summary for finding the corresponding scientific article (long)

Two novel GANs are constructed to generate high-quality 3D fMRI brain images and synthetic brain images greatly help to improve downstream classification tasks.

We introduce an alternative GAN design based on random routes in generator, which can serve as a tool for generative models interpretability.