Question: represent the text to find a very short summary (1-2 sentences max)

Question: In this paper, we describe the "implicit autoencoder" (IAE), a generative autoencoder in which both the generative path and the recognition path are parametrized by implicit distributions. We use two generative adversarial networks to define the reconstruction and the regularization cost functions of the implicit autoencoder, and derive the learning rules based on maximum-likelihood learning. Using implicit distributions allows us to learn more expressive posterior and conditional likelihood distributions for the autoencoder. Learning an expressive conditional likelihood distribution enables the latent code to only capture the abstract and high-level information of the data, while the remaining information is captured by the implicit conditional likelihood distribution. For example, we show that implicit autoencoders can disentangle the global and local information, and perform deterministic or stochastic reconstructions of the images. We further show that implicit autoencoders can disentangle discrete underlying factors of variation from the continuous factors in an unsupervised fashion, and perform clustering and semi-supervised learning. Deep generative models have achieved remarkable success in recent years. One of the most successful models is the generative adversarial network (GAN) BID7, which employs a two player min-max game. The generative model, G, samples the noise vector z ∼ p(z) and generates the sample G(z). The discriminator, D(x), is trained to identify whether a point x comes from the data distribution or the model distribution; and the generator is trained to maximally confuse the discriminator. The cost function of GAN is DISPLAYFORM0 GANs can be viewed as a general framework for learning implicit distributions BID18 BID12. Implicit distributions are probability distributions that are obtained by passing a noise vector through a deterministic function that is parametrized by a neural network. In the probabilistic machine learning problems, implicit distributions trained with the GAN framework can learn distributions that are more expressive than the tractable distributions trained with the maximum-likelihood framework. Variational autoencoders (VAE) BID13 BID20 are another successful generative models that use neural networks to parametrize the posterior and the conditional likelihood distributions. Both networks are jointly trained to maximize a variational lower bound on the data log-likelihood. One of the limitations of VAEs is that they learn factorized distributions for both the posterior and the conditional likelihood distributions. In this paper, we propose the "implicit autoencoder" (IAE) that uses implicit distributions for learning more expressive posterior and conditional likelihood distributions. Learning a more expressive posterior will in a tighter variational bound; and learning a more expressive conditional likelihood distribution will in a global vs. local decomposition of information between the prior and the conditional likelihood. This enables the latent code to only capture the information that we care about such as the high-level and abstract information, while the remaining low-level information of data is separately captured by the noise vector of the implicit decoder. Implicit distributions have been previously used in learning generative models in works such as adversarial autoencoders (AAE) BID16, adversarial variational Bayes (AVB) , ALI , BiGAN BID5 and other works such as BID12 BID22. The global vs. local decomposition of information has also been studied in previous works such as PixelCNN autoencoders (van den), PixelVAE BID9, variational lossy autoencoders BID4, PixelGAN autoencoders BID15, or other works such as BID2 BID8 BID0. In the next section, we first propose the IAE and then establish its connections with the related works. Let x be a datapoint that comes from the data distribution p data (x). The encoder of the implicit autoencoder (FIG0) defines an implicit variational posterior distribution q(z|x) with the function z = f φ (x,) that takes the input x along with the input noise vector and outputsẑ. The decoder of the implicit autoencoder defines an implicit conditional likelihood distribution p(x|z) with the functionx = g θ (ẑ, n) that takes the codeẑ along with the latent noise vector n and outputs a reconstruction of the imagex. In this paper, we refer toẑ as the latent code or the global code, and refer to the latent noise vector n as the local code. Let p(z) be a fixed prior distribution, p(x, z) = p(z)p(x|z) be the joint model distribution, and p(x) be the model distribution. The variational distribution q(z|x) induces the joint data distribution q(x, z), the aggregated posterior distribution q(z), and the inverse posterior/encoder distribution q(x|z) as follows: DISPLAYFORM0 Maximum likelihood learning is equivalent to matching the model distribution p(x) to the data distribution p data (x); and learning with variational inference is equivalent to matching the joint model distribution p(x, z) to the joint data distribution q(x, z). The entropy of the data distribution H data (x), the entropy of the latent code H(z), the mutual information I(x; z), and the conditional entropies H(x|z) and H(z|x) are all defined under the joint data distribution q(x, z) and its marginals p data (x) and q(z). Using the aggregated posterior distribution q(z), we can define the joint reconstruction distribution r(x, z) and the aggregated reconstruction distribution r(x) as follows: DISPLAYFORM1 Note that in general we have r(x, z) = q(x, z) = p(x, z), q(z) = p(z), and r(x) = p data (x) = p(x).We now use different forms of the aggregated evidence lower bound (ELBO) to describe the IAE and establish its connections with VAEs and AAEs. DISPLAYFORM2 Mutual Info. DISPLAYFORM3 Entropy of Data DISPLAYFORM4 Entropy of Data See Appendix A for the proof. The standard formulation of the VAE (Equation 4) only enables us to learn factorized Gaussian posterior and conditional likelihood distributions. The AAE BID16 (Equation 5) and the AVB BID17 enable us to learn implicit posterior distributions, but their conditional likelihood distribution is still a factorized Gaussian distribution. However, the IAE enables us to learn implicit distributions for both the posterior and the conditional likelihood distributions. Similar to VAEs and AAEs, the IAE (Equation 6) has a reconstruction cost function and a regularization cost function, but trains each of them with a GAN. The IAE reconstruction cost is E z∼q(z) KL(q(x|z) p(x|z)). The standard VAE uses a factorized decoder, which has a very limited stochasticity. Thus, the standard VAE performs almost deterministic reconstructions by learning to invert the deterministic mapping of the encoder. The IAE, however, uses a powerful implicit decoder to perform stochastic reconstructions, by learning to match the expressive decoder distribution p(x|z) to the inverse encoder distribution q(x|z). We note that there are other variants of VAEs that can also learn expressive decoder distributions by using autoregressive neural networks. We will discuss these models later in this section. Equation 8 contrasts the reconstruction cost of standard autoencoders that is used in VAEs/AAEs, with the reconstruction cost of IAEs. DISPLAYFORM5 We can see from Equation 8 that similar to IAEs, the reconstruction cost of the autoencoder encourages matching the decoder distribution to the inverse encoder distribution. But in autoencoders, the cost function also encourages minimizing the conditional entropy H(x|z), or maximizing the mutual information I(x, z). Maximizing the mutual information in autoencoders enforces the latent code to capture both the global and local information. In contrast, in IAEs, the reconstruction cost does not penalize the encoder for losing the local information, as long as the decoder can invert the encoder distribution. In order to minimize the reconstruction cost function of the IAE, we re-write it in the form of a distribution matching cost function between the joint data distribution and the joint reconstruction distribution KL(q(x, z) r(x, z)) (Equation 7). This KL divergence is approximately minimized with the reconstruction GAN. The IAE has also a regularization cost function KL(q(z) p(z)) that matches the aggregated posterior distribution with a fixed prior distribution. This is the same regularization cost function used in AAEs (Equation 5), and is approximately minimized with the regularization GAN. Note that the last term in FIG4 is the entropy of the data distribution that is fixed. Training Process. We now describe the training process. We pass a given point x ∼ p data (x) through the encoder and the decoder to obtainẑ ∼ q(z) andx ∼ r(x). We now train the discriminator of the reconstruction GAN to identify the positive example (x,ẑ) from the negative example (x,ẑ). Suppose this discriminator function at its optimality is D * (x, z). We try to confuse this discriminator by backpropagating through the negative example (x,ẑ) 1 and updating the encoder and decoder weights. More specifically, the generative loss of the reconstruction GAN is T * (x, z) = − log D * (x, z), which defines the reconstruction cost of the IAE. We use the re-parametrization trick to update the encoder and decoder weights by computing the unbiased Monte Carlo estimate of the gradient of the reconstruction cost T * (x, z) with respect to (φ, θ) as follows: DISPLAYFORM6 We call this process the adversarial reconstruction. Similarly, we train the discriminator of the regularization GAN to identify the positive example z ∼ p(z) from the negative exampleẑ ∼ q(z). This discriminator now defines the regularization cost function, which can provide us with a gradient to update only the encoder weights. We call this process the adversarial regularization. Optimizing the adversarial regularization and reconstruction cost functions encourages p(x|z) = q(x|z) and p(z) = q(z), which in the model distribution capturing the data distribution p(x) = p data (x).We note that in this work, we use the original formulation of GANs BID7 to match the distributions. As a , the gradient that we obtain from the adversarial training, only approximately follows the gradient of the variational bound on the data log-likelihood. However, as shown in BID19, the objective of the GAN can be modified to optimize any f -divergence including the KL divergence. Bits-Back Interpretation of the IAE Objective. In Appendix B, we describe an information theoretic interpretation of the ELBO of IAEs (Equation 7) using the Bits-Back coding argument BID10 BID4 BID8. In IAEs, the dimension of the latent vector along with its prior distribution defines the capacity of the latent code, and the dimension of the latent noise vector along with its distribution defines the capacity of the implicit decoder. By adjusting these dimensions and distributions, we can have a full control over the decomposition of information between the latent code and the implicit decoder. In one extreme case, by removing the noise vector, we can have a fully deterministic autoencoder that captures all the information by its latent code. In the other extreme case, we can remove the global latent code and have an unconditional implicit distribution that can capture the whole data distribution by itself. The global vs. local decomposition of information in IAEs is further discussed in Appendix C from an information theoretic perspective. In IAEs, we can choose to only optimize the reconstruction cost or both the reconstruction and the regularization costs. In the following, we discuss four special cases of the IAE and establish connections with the related methods. In this case, we remove the noise vectors from the IAE, which makes both q(z|x) and p(x|z)deterministic. We then only optimize the reconstruction cost E z∼q(z) KL(q(x|z) p(x|z)). As a , similar to the standard autoencoder, the deterministic decoder p(x|z) learns to match to the inverse deterministic encoder q(x|z), and thus the IAE learns to perform exact and deterministic reconstruction of the original image, while the latent code is learned in an unconstrained fashion. In other words, in standard autoencoders, the Euclidean cost explicitly encouragesx to reconstruct x, and in case of uncertainty, performs mode averaging by blurring the reconstructions; however, in IAEs, the adversarial reconstruction implicitly encouragesx to reconstruct x, and in case of uncertainty, captures this uncertainty by the local noise vector (Case 3), which in sharp reconstructions. In the previous case, the latent code was learned in an unconstrained fashion. We now keep the decoder deterministic and add the regularization term which matches the aggregated posterior distribution to a fixed prior distribution. In this case, the IAE reduces to the AAE with the difference that the IAE performs adversarial reconstruction rather than Euclidean reconstruction. This case of the IAE defines a valid generative model where the latent code captures all the information of the data distribution. In order to sample from this model, we first sample from the imposed prior p(z) and then pass this sample through the deterministic decoder. In this case of the IAE, we only optimize KL(q(x, z) r(x, z)), while p(x|z) is a stochastic implicit distribution. Matching the joint distribution q(x, z) to r(x, z) ensures that their marginal distributions would also match; that is, the aggregated reconstruction distribution r(x) matches the data distribution p data (x). This model by itself defines a valid generative model in which both the prior, which in this case is q(z), and the conditional likelihood p(x|z) are learned at the same time. In order to sample from this generative model, we initially sample from q(z) by first sampling a point x ∼ p data (x) and then passing it through the encoder to obtain the latent codeẑ ∼ q(z). Then we sample from the implicit decoder distribution conditioned onẑ to obtain the stochastic reconstructionx ∼ r(x). If the decoder is deterministic (Case 1), the reconstructionx would be the same as the original image x. But if the decoder is stochastic, the global latent code only captures the abstract and high-level information of the image, and the stochastic reconstructionx only shares this high-level information with the original x. This case of the IAE is related to the PixelCNN autoencoder BID23, where the decoder is parametrized by an autoregressive neural network which can learn expressive distributions, while the latent code is learned in an unconstrained fashion. In the previous case, we showed that even without the regularization term, r(x) will capture the data distribution. But the main drawback of the previous case is that its prior q(z) is not a parametric distribution that can be easily sampled from. One way to fix this problem is to fit a parametric prior p(z) to q(z) once the training is complete, and then use p(z) to sample from the model. However, a better solution would be to consider a fixed and pre-defined prior p(z), and impose it on q(z) during the training process. Indeed, this is the regularization term that the ELBO suggests in Equation 7. By adding the adversarial regularization cost function to match q(z) to p(z), we ensure that r(x) = p data (x) = p(x). Now sampling from this model only requires first sampling from the pre-defined prior z ∼ p(z), and then sampling from the conditional implicit distribution to obtain x ∼ r(x). In this case, the information of data distribution is captured by both the fixed prior and the learned conditional likelihood distribution. Similar to the previous case, the latent code captures the high-level and abstract information, while the remaining local and low-level information is captured by the implicit decoder. We will empirically show this decomposition of information on different datasets in Section 2.1.1 and Section 2.1.2. This decomposition of information has also been studied in other works such as PixelVAE BID9, variational lossy autoencoders BID4, PixelGAN autoencoders BID15 and variational Seq2Seq autoencoders BID2. However, the main drawback of these methods is that they all use autoregressive decoders which are not parallelizable, and are much more computationally expensive to scale up than the implicit decoders. Another advantage of implicit decoders to autoregressive decoders is that in implicit decoders, the local statistics is captured by the local code representation; but in autoregressive decoders, we do not learn a vector representation for the local statistics. Connections with ALI and BiGAN. In ALI BID6 and BiGAN BID5 models, there are two separate networks that define the joint data distribution q(x, z) and the joint model distribution p(x, z). The parameters of these networks are trained using the gradient that comes from a single GAN that tries to match these two distributions. However, in the IAE, similar to VAEs or AAEs, the encoder and decoder are stacked on top of each other and trained jointly. So the gradient that the encoder receives comes through the decoder and the conditioning vector. In other words, in the ALI model, the input to the conditional likelihood is the samples of the prior distribution, whereas in the IAE, the input to the conditional likelihood is the samples of the variational posterior distribution, while the prior distribution is separately imposed on the aggregated posterior distribution by the regularization GAN. This makes the training dynamic of IAEs similar to that of autoencoders, which encourages better reconstructions. Recently, many variants of ALI have been proposed for improving its reconstruction performance. For example, the HALI BID1 ) uses a Markovian generator to achieve better reconstructions, and ALICE BID14 augments the ALI's cost by a joint distribution matching cost function between (x,x) and (x, x), which is different from our reconstruction cost. In this section, we show that the IAE can learn a global vs. local decomposition of information between the latent code and the implicit decoder. We use the Gaussian distribution for both the global and local codes, and show that by adjusting the dimensions of the global and local codes, we can have a full control over the decomposition of information. FIG1 shows the performance of the IAE on the MNIST dataset. By removing the local code and using only a global code of size 20D FIG1, the IAE becomes a deterministic autoencoder. In this case, the global code of the IAE captures all the information of the data distribution and the IAE achieves almost perfect reconstructions. By decreasing the global code size to 10D and using a 100D local code FIG1 ), the global code retains the global information of the digits such as the label information, while the local code captures small variations in the style of the digits. By using a smaller global code of size 5D FIG1 ), the encoder loses more local information and thus the global code captures more abstract information. For example, we can see from FIG1 that the encoder maps visually similar digits such as {3, 5, 8} or {4, 9} to the same global code, while the implicit decoder learns to invert this mapping and generate stochastic reconstructions that share the same high-level information with the original images. Note that if we completely remove the global code, the local code captures all the information, similar to the standard unconditional GAN. Figure 3 shows the performance of the IAE on the SVHN dataset. When using a 150D global code with no local code (Figure 3b), similar to the standard autoencoder, the IAE captures all the information by its global code and can achieve almost perfect reconstructions. However, when using a 75D global code along with a 1000D local code (Figure 3c), the global code of the IAE only captures the middle digit information as the global information, and loses the left and right digit information. At the same time, the implicit decoder learns to invert the encoder distribution by keeping the middle digit and generating synthetic left and right SVHN digits with the same style of the middle digit. Figure 4 shows the performance of the IAE on the CelebA dataset. When using a 150D global code with no local code (Figure 4b), the IAE achieves almost perfect reconstructions. But when using a 50D global code along with a 1000D local code (Figure 4c), the global code of the IAE only retains the global information of the face such as the general shape of the face, while the local code captures the local attributes of the face such as eyeglasses, mustache or smile. In IAEs, by using a categorical global code along with a Gaussian local code, we can disentangle the discrete and continuous factors of variation, and perform clustering and semi-supervised learning. Clustering. In order to perform clustering with IAEs, we change the architecture of FIG0 by using a softmax function in the last layer of the encoder, as a continuous relaxation of the categorical global code. The dimension of the categorical code is the number of categories that we wish the data to be clustered into. The regularization GAN is trained directly on the continuous output probabilities of the softmax simplex, and imposes the categorical distribution on the aggregated posterior distribution. This adversarial regularization imposes two constraints on the encoder output. The first constraint is that the encoder has to make confident decisions about the cluster assignments. The second constraint is that the encoder must distribute the points evenly across the clusters. As a , the global code only captures the discrete underlying factors of variation such as class labels, while the rest of the structure of the image is separately captured by the Gaussian local code of the implicit decoder. Figure 5 shows the samples of the standard GAN and the IAE trained on the mixture of Gaussian data. Figure 5b shows the samples of the GAN, which takes a 7D categorical and a 10D Gaussian noise vectors as the input. Each sample is colored based on the one-hot noise vector that it was generated from. We can see that the GAN has failed to associate the categorical noise vector to different mixture components, and generate the whole data solely by using its Gaussian noise vector. Ignoring the categorical noise forces the GAN to do a continuous interpolation between different mixture components, which in reducing the quality of samples. Figure 5c shows the samples of the IAE whose implicit decoder architecture is the same as the GAN. The IAE has a 7D categorical global code (inferred by the encoder) and a 10D Gaussian noise vector. In this case, the inference network of the IAE learns to cluster the data in an unsupervised fashion, while its generative path learns to condition on the inferred cluster labels and generate each mixture component using the stochasticity of the Gaussian noise vector. This example highlights the importance of using discrete latent variables for improving generative models. A related work is the InfoGAN BID3, which uses a reconstruction cost in the code space to prevent the GAN from ignoring the categorical noise vector. The relationship of InfoGANs with IAEs is discussed in details in Section 3. The style (local noise vector) is drawn from a Gaussian distribution and held fixed across each row. FIG3 shows the clustering performance of the IAE on the MNIST dataset. The IAE has a 30D categorical global latent code and a 10D Gaussian local code. Each column corresponds to the conditional samples from one of the learned clusters (only 20 are shown). The local code is sampled from the Gaussian distribution and held fixed across each row. We can see that the discrete global latent code of the network has learned discrete factors of variation such as the digit identities, while the writing style information is separately captured by the continuous Gaussian noise vector. This network obtains about 5% error rate in classifying digits in an unsupervised fashion, just by matching each cluster to a digit type. Semi-Supervised Learning. The IAE can be used for semi-supervised classification. In order to incorporate the label information, we set the number of clusters to be the same as the number of class labels and additionally train the encoder weights on the labeled mini-batches to minimize the cross-entropy cost. On the MNIST dataset with 100 labels, the IAE achieves the error rate of 1.40%. In comparison, the AAE achieves 1.90%, and the Improved-GAN BID21 achieves 0.93%. On the SVHN dataset with 1000 labels, the IAE achieves the error rate of 9.80%. In comparison, the AAE achieves 17.70%, and the Improved-GAN achieves 8.11%. In this section, we describe the "Flipped Implicit Autoencoder" (FIAE), which is a generative model that is very closely related to IAEs. Let z be the latent code that comes from the prior distribution p(z). The encoder of the FIAE FIG4 ) parametrizes an implicit distribution that uses the noise vector n to define the conditional likelihood distribution p(x|z). The decoder of the FIAE parametrizes an implicit distribution that uses the noise vector to define the variational posterior distribution q(z|x). In addition to the distributions defined in Section 2, we also define the joint latent reconstruction distribution s(x, z), and the aggregated latent reconstruction distribution s(z) as follows: DISPLAYFORM0 The objective of the standard variational inference is minimizing KL(q(x, z) p(x, z)), which is the variational upper-bound on KL(p data (x) p(x)). The objective of FIAEs is the reverse KL divergence KL(p(x, z) q(x, z)), which is the variational upper-bound on KL(p(x) p data (x)). The FIAE optimizes this variational bound by splitting it into a reconstruction term and a regularization term as follow: DISPLAYFORM1 Cond. Entropy DISPLAYFORM2 where the conditional entropy H(z|x) is defined under the joint model distribution p(x, z). Similar to the IAE, the FIAE has a regularization term and a reconstruction term (Equation 14 and Equation 15). The regularization cost uses a GAN to train the encoder (conditional likelihood) such that the model distribution p(x) matches the data distribution p data (x). The reconstruction cost uses a GAN to train both the encoder (conditional likelihood) and the decoder (variational posterior) such that the joint model distribution p(x, z) matches the joint latent reconstruction distribution s(x, z).Connections with ALI and BiGAN. In ALI BID6 and BiGAN BID5 ) models, the input to the recognition network is the samples of the real data p data (x); however, in FIAEs, the recognition network only gets to see the synthetic samples that come from the simulated data p(x), while at the same time, the regularization cost ensures that the simulated data distribution is close the real data distribution. Training the recognition network on the simulated data in FIAEs is in spirit similar to the "sleep" phase of the wake-sleep algorithm BID11, during which the recognition network is trained on the samples that the network "dreams" up. One of the flaws of training the recognition network on the simulated data is that early in the training, the simulated data do not look like the real data, and thus the recognition path learns to invert the generative path in part of the data space that is far from the real data distribution. As the , the reconstruction GAN might not be able to keep up with the moving simulated data distribution and get stuck in a local optimum. However, in our experiments with FIAEs, we did not find this to be a major problem. Connections with InfoGAN. InfoGANs BID3, similar to FIAEs, train the variational posterior network on the simulated data; however, as shown in Equation 13, InfoGANs use an explicit reconstruction cost function (e.g., Euclidean cost) on the code space for learning the variational posterior. In order to compare FIAEs and InfoGANs, we train them on a toy dataset with four data-points and use a 2D Gaussian prior (Figure 8 and Figure 9). Each colored cluster corresponds to the posterior distribution of one data-point. In InfoGANs, using the Euclidean cost to reconstruct the code corresponds to learning a factorized Gaussian variational posterior distribution (Figure 8b) 2. This constraint on the variational posterior restricts the family of the conditional likelihoods that the model can learn by enforcing the generative path to learn a conditional likelihood whose true posterior could fit to the factorized Gaussian approximation of the posterior. For example, we can see in Figure 8a that the model has learned a conditional likelihood whose true posterior is axis-aligned, so that it could better match the factorized Gaussian variational posterior (Figure 8b). In contrast, the FIAE can learn an arbitrarily expressive variational posterior distribution (Figure 9b), which enables the generative path to learn a more expressive conditional likelihood and true posterior (Figure 9a).One of the main flaws of optimizing the reverse KL divergence is that the variational posterior will have the mode-covering behavior rather than the mode-picking behavior. For example, we can see from Figure 8b that the Gaussian posteriors of different data-points in InfoGAN have some overlap; but this is less of a problem in the FIAE (Figure 9b), as it can learn a more expressive q(z|x). This mode-averaging behavior of the posterior can be also observed in the wake-sleep algorithm, in which during the sleep phase, the recognition network is trained using the reverse KL divergence objective. The FIAE objective is not only an upper-bound on KL(p(x) p data (x)), but is also an upper-bound on KL(p(z) q(z)) and KL(p(z|x) q(z|x)). As a , the FIAE matches the variational posterior q(z|x) to the true posterior p(z|x), and also matches the aggregated posterior q(z) to the prior p(z). For example, we can see in Figure 9b that q(z) is very close to the Gaussian prior. However, the InfoGAN objective is theoretically not an upper-bound on KL(p(x) p data (x)), KL(p(z) q(z)) or KL(p(z|x) q(z|x)). As a , in InfoGANs, the variational posterior q(z|x) need not be close to the true posterior p(z|x), or the aggregated posterior q(z) does not have to match the prior p(z). Reconstruction. In this section, we show that the variational posterior distribution of the FIAE can invert its conditional likelihood function by showing that the network can perform reconstructions of the images. We make both the conditional likelihood and the variational posterior deterministic by removing both noise vectors n and. FIG0 shows the performance of the FIAE with a code size of 15 on the test images of the MNIST dataset. The reconstructions are obtained by first passing the image through the recognition network to infer its latent code, and then using the inferred latent code at the input of the conditional likelihood to generate the reconstructed image. Clustering. Similar to IAEs, we can use FIAEs for clustering. We perform an experiment on the MNIST dataset by choosing a discrete categorical latent code z of size 10, which captures the digit identity; and a continuous Gaussian noise vector n of size 10, which captures the style of the digit. The variational posterior distribution q(z|x) is also parametrized by an implicit distribution with a Gaussian noise vector of size 20, and performs inference only over the digit identity z. Once the network is trained, we can use the variational posterior to cluster the test images of the MNIST dataset. This network achieves the error rate of about 2% in classifying digits in an unsupervised fashion by matching each categorical code to a digit type. We observed that when there is uncertainty in the digit identity, different draws of the noise vector in different one-hot vectors at the output of the recognition network, showing that the implicit decoder can efficiently capture the uncertainty. In this paper, we proposed the implicit autoencoder, which is a generative autoencoder that uses implicit distributions to learn expressive variational posterior and conditional likelihood distributions. We showed that in IAEs, the information of the data distribution is decomposed between the prior and the conditional likelihood. When using a low dimensional Gaussian distribution for the global code, we showed that the IAE can disentangle high-level and abstract information from the low-level and local statistics. We also showed that by using a categorical latent code, we can learn discrete factors of variation and perform clustering and semi-supervised learning. In this section, we describe an information theoretic interpretation of the ELBO of IAEs (Equation 7) using the Bits-Back coding argument BID10 BID4 BID8. Maximizing the variational lower bound is equivalent to minimizing the expected description length of a source code for the data distribution p data (x) when the code is designed under the model distribution p(x). In order to transmit x, the sender uses a two-part code. It first transmits z, which ideally would only require H(z) bits; however, since the code is designed under p(z), the sender has to pay the penalty of KL(q(z) p(z)) extra bits to compensate for the mismatch between q(z) and p(z). After decoding z, the receiver now has to resolve the uncertainty of q(x|z) in order to reconstruct x, which ideally requires the sender to transmit the second code of the length H(x|z) bits. However, since the code is designed under p(x|z), the sender has to pay the penalty of E z∼q(z) KL(q(x|z) p(x|z)) extra bits on average to compensate for the fact that the conditional decoder p(x|z) has not perfectly captured the inverse encoder distribution q(x|z); i.e., the autoencoder has failed to achieve perfect stochastic reconstruction. But for a given x, the sender could use the stochasticity of q(z|x) to encode other information. Averaged over the data distribution, this would get the sender H(z|x) "bits back" that needs to be subtracted in order to find the true cost for transmitting x: DISPLAYFORM0 From Equation 25, we can see that the IAE only minimizes the extra number of bits required for transmitting x, while the VAE minimizes the total number of bits required for the transmission. Continuous Variables. The Bits-Back argument is also applicable to continuous random variables. Suppose x and z are real-valued random variables. Let h(x) and h(z) be the differential entropies of x and z; and H(x) and H(z) be the discrete entropies of the quantized versions of x and z, with the quantization interval of ∆x and ∆z. We have DISPLAYFORM1 The sender first transmits the real-valued random variable z, which requires transmission of H(z) = h(z) − log ∆z bits, as well as KL(q(z) p(z)) extra bits. As ∆z → 0, we will have H(z) → ∞, which, as expected, implies that the sender would need infinite number of bits to source code and send the real-valued random variable z. However, as we shall see, we are going to get most of these bits back from the receiver at the end. After the first message, the sender then sends the second message, which requires transmission of h(x|z) − log ∆x bits, as well as E z∼q(z) KL(q(x|z) p(x|z)) extra bits. Once the receiver decodes z, and form that decodes x, it can decode a secondary message of the average length H(z|x) = h(z|x) − log ∆z, which needs to be subtracted in order to find the true cost for transmitting x: DISPLAYFORM2 From Equation 28, we can interpret the IAE cost as the extra number of bits required for the transmission of x. In IAEs, the global code (prior) captures the global information of data, while the remaining local information is captured by the local noise vector (conditional likelihood). In this section, we describe the global vs. local decomposition of information from an information theoretic perspective. In order to transmit x, the sender first transmits z and then transmits the residual bits required for reconstructing x, using a source code that is designed based on p(x, z). If p(z) and p(x|z) are powerful enough, in theory, they can capture any q(x, z), and thus regardless of the decomposition of information, the sender would only need to send H data (x) bits. In this case, the ELBO does not prefer one decomposition of information to another. But if the capacities of p(z) and p(x|z) are limited, the sender will have to send extra bits due to the distribution mismatch, ing in the regularization and reconstruction errors. But now different decompositions of information will in different numbers of extra bits. So the sender has to decompose the information in a way that is compatible with the source codes that are designed based on p(z) and p(x|z). The prior p(z) that we use in this work is a low-dimensional Gaussian or categorical distribution. So the regularization cost encourages the sender to encode low-dimensional or simple concepts in z that is consistent with p(z); otherwise, the sender would need to pay a large cost for KL(q(z) p(z)). The choice of the information encoded in z would also affect the extra number of bits of E z∼q(z) KL(q(x|z) p(x|z)), which is the reconstruction cost. This is because the conditional decoder p(x|z) with its limited capacity is supposed to capture the inverse encoder distribution q(x|z). So the sender must encode the kind of information in z that after being observed, can maximally remove the stochasticity of q(x|z) so as to lower the burden on p(x|z) for matching to q(x|z). So the reconstruction cost encourages learning the kind of concepts that can remove as much uncertainty as possible from the data distribution. By balancing the regularization and reconstruction costs, the latent code learns global concepts which are low-dimensional or simple concepts that can maximally remove uncertainty from data. Examples of global concepts are digit identities in the MNIST dataset, objects in natural images or topics in documents. There are two methods to implement how the reconstruction GAN conditions on the global code. Location-Dependent Conditioning. Suppose the size of the first convolutional layer of the discriminator is (batch, width, height, channels). We use a one layer neural network with 1000 ReLU hidden units to transform the global code of size (batch, global_code_size) to a spatial tensor of size (batch, width, height, 1). We then broadcast this tensor across the channel dimension to get a tensor of size (batch, width, height, channels), and then add it to the first layer of the discriminator as an adaptive bias. In this method, the latent vector has spatial and location-dependent information within the feature map. This is the method that we used in deterministic and stochastic reconstruction experiments. Location-Invariant Conditioning. Suppose the size of the first convolutional layer of the discriminator is (batch, width, height, channels). We use a linear mapping to transform the global code of size (batch, global_code_size) to a tensor of size (batch, channels). We then broadcast this tensor across the width and height dimensions, and then add it to the first layer of the discriminator as an adaptive bias. In this method, the global code is encouraged to learn the global information that is location-invariant such as the class label information. We used this method in all the clustering and semi-supervised learning experiments. The regularization discriminator in all the experiments is a two-layer neural network, where each layer has 2000 hidden units with the ReLU activation function. The architecture of the encoder, the decoder and the reconstruction discriminator for each dataset is as follows.

Represent the short summary for finding the corresponding scientific article (long)

We propose a generative autoencoder that can learn expressive posterior and conditional likelihood distributions using implicit distributions, and train the model using a new formulation of the ELBO.

We show that posterior collapse in linear VAEs is caused entirely by marginal log-likelihood (not ELBO). Experiments on deep VAEs suggest a similar phenomenon is at play.


Question: Represent the scientific article to find a short TLDR

Question: Strong inductive biases allow children to learn in fast and adaptable ways. Children use the mutual exclusivity (ME) bias to help disambiguate how words map to referents, assuming that if an object has one label then it does not need another. In this paper, we investigate whether or not standard neural architectures have a ME bias, demonstrating that they lack this learning assumption. Moreover, we show that their inductive biases are poorly matched to lifelong learning formulations of classification and translation. We demonstrate that there is a compelling case for designing neural networks that reason by mutual exclusivity, which remains an open challenge. Children are remarkable learners, and thus their inductive biases should interest machine learning researchers. To help learn the meaning of new words efficiently, children use the "mutual exclusivity" (ME) bias -the assumption that once an object has one name, it does not need another (Figure 1). In this paper, we examine whether or not standard neural networks demonstrate the mutual exclusivity bias, either as a built-in assumption or as a bias that develops through training. Moreover, we examine common benchmarks in machine translation and object recognition to determine whether or not a maximally efficient learner should use mutual exclusivity. The mutual exclusivity task used in cognitive development research . Children tend to associate the novel word ("dax") with the novel object (right). When children endeavour to learn a new word, they rely on inductive biases to narrow the space of possible meanings. Children learn an average of about 10 new words per day from the age of one until the end of high school , a feat that requires managing a tractable set of candidate meanings. A typical word learning scenario has many sources of ambiguity and uncertainty, including ambiguity in the mapping between words and referents. Children hear multiple words and see multiple objects within a single scene, often without clear supervisory signals to indicate which word goes with which object . The mutual exclusivity assumption helps to resolve ambiguity in how words maps to their referents. examined scenarios like Figure 1 that required children to determine the referent of a novel word. For instance, children who know the meaning of "cup" are presented with two objects, one which is familiar (a cup) and another which is novel (an unusual object). Given these two objects, children are asked to "Show me a dax," where "dax" is a novel nonsense word. Markman and Wachtel found that children tend to pick the novel object rather than the familiar object. Although it is possible that the word "dax" could be another word for referring to cups, children predict that the novel word refers to the novel object -demonstrating a "mutual exclusivity" bias that familiar objects do not need another name. This is only a preference; with enough evidence, children must eventually override this bias to learn hierarchical categories: a Dalmatian can be called a "Dalmatian," a "dog", or a "mammal" . As an often useful but sometimes misleading cue, the ME bias guides children when learning the words of their native language. It is instructive to compare word learning in children and machines, since word learning is also a widely studied problem in machine learning and artificial intelligence. There has been substantial (a) (b) Figure 2: Evaluating mutual exclusivity in a feedforward (a) and seq2seq (b) neural network. (a) After training on a set of known objects, a novel label ("dax") is presented as a one-hot input vector. The network maps this vector to a one-hot output vector representing the predicted referent, through an intermediate embedding layer and an optional hidden layer (not shown). A representative output vector produced by a trained network is shown, placing almost all of the probability mass on known outputs. (b) A similar setup for mapping sequences of labels to their referents. During the test phase a novel label "dax" is presented and the ME Score at that output position is computed. recent progress in object recognition, much of which is attributed to the success of deep neural networks and the availability of very large datasets . But when only one or a few examples of a novel word are available, deep learning algorithms lack human-like sample efficiency and flexibility . Insights from cognitive science and cognitive development can help bridge this gap, and ME has been suggested as a psychologically-informed assumption relevant to machine learning. In this paper, we examine standard neural networks to understand if they have an ME bias. Moreover, we analyze whether or not ME is a good assumption in lifelong variants of common translation and object recognition tasks. Children utilize a variety of inductive biases like mutual exclusivity when learning the meaning of words . Previous work comparing children and neural networks has focused on the shape bias -an assumption that objects with the same name tend to have the same shape, as opposed to color or texture . Children acquire a shape bias over the course of language development , and neural networks can do so too, as shown in synthetic learning scenarios and large-scale object recognition tasks (see also and for alternative findings). This bias is related to how quickly children learn the meaning of new words , and recent findings also show that guiding neural networks towards the shape bias improves their performance . In this work, we take initial steps towards a similar investigation of the ME bias in neural networks. Compared to the shape bias, ME has broader implications for machine learning systems; as we show in our analyses, the bias is relevant beyond object recognition. Closer to the present research, analyzed an ME-like effect in neural machine translation systems at the sentence level, rather than the word level considered in developmental studies and our analyses here. showed that neural machine translation systems often learn many-to-one sentence mappings that in meaning loss, such that two different sentences (meanings) in the source language are mapped to the same sentence (meaning) in the target language. Using a trained network, they show how a probabilistic pragmatics model can be used as a post-processor to preserve meaning and encourage one-to-one mappings. These sentence-level biases do not necessarily indicate how models behave at the word level, and we are interested in the role of ME during learning rather than as a postprocessing step. Nevertheless, Cohn-Gordon and Goodman's are important and encouraging, raising the possibility that ME could aid in training deep learning systems. In this section, we investigate whether or not standard neural network models have a mutual exclusivity bias. Paralleling the developmental paradigm , ME is analyzed by presenting a novel stimulus ("Show me the dax") and asking models to predict which outputs (meanings) are most likely. The strength of the bias is operationalized as the aggregate probability mass placed on the novel rather than the familiar meanings. Our analyses relate to classic experiments by Marcus on whether neural networks can generalize outside their training space (; . Marcus showed that a feedforward autoencoder trained on arbitrary binary patterns fails to generalize to an output unit that was never activated during training. Our aim is to study whether standard architectures can recognize and learn a more abstract pattern -a perfect one-to-one mapping between input symbols and output symbols. Specifically, we are interested in model predictions regarding unseen meanings given a novel input. We also test for ME using modern neural networks in two settings using synthetic data: classification (feedforward classifiers) and translation (sequence-to-sequence models; as reported in Appendix A). Synthetic data. We consider a simple one-to-one mapping task inspired by. Translating this into a synthetic experiment, input units denote words and output units denote objects. Thus, the dataset consists of 100 pairs of input and output patterns, each of which is a one-hot vector of length 100. Each input vector represents a label (e.g., 'hat', 'cup', 'dax') and each output vector represents a possible referent object (meaning). Figure 2a shows the input and output patterns for the'dax' case, and similar patterns are defined for the other 99 input and output symbols. A one-to-one correspondence between each input symbol and each output symbol is generated through a random permutation, and there is no structure to the data beyond the arbitrary one-to-one relationship. Models are trained on 90 name-referent pairs and evaluated on the remaining 10 test pairs. No model can be expected to know the correct meaning of each test name -there is no way to know from the training data -but several salient patterns are discoverable. First, there is a precise one-toone relationship exemplified by the 90 training items; the 10 test items can be reasonably assumed to follow the same one-to-one pattern, especially if the network architecture has exactly 10 unused input symbols and 10 unused output symbols. Second, the perfect one-to-one relationship ensures a perfect ME bias in the structure of the data. Although the learner does not know precisely which new output symbol a new input symbol refers to, it should predict that the novel input symbol will correspond to one of the novel output symbols. An ideal learner should discover that an output unit with a known label does not need another -in other words, it should utilize ME to make predictions. Mutual exclusivity. We ask the neural network to "Show me the dax" by activating the "dax" input unit and asking it to select amongst possible referents (similar to Figure 1). The network produces a probability distribution over candidate referents (see Figure 2a), and can make relative (two object) comparisons by isolating the two relevant scores. To quantify the overall propensity toward ME, we define an "ME score" that measures the aggregate probability assigned to all of the novel output symbols as opposed to familiar outputs, corresponding to better performance on the classic forced choice ME task. Let us denote the training symbol by Y, drawn from the data distribution (X, Y) ∼ D and the held out symbols Y drawn from (X, Y) ∼ D. The mutual exclusivity score is the sum probability assigned to unseen output symbols Y when shown a novel input symbol x ∈ X ME Score = 1 averaged over each of the test items. An ideal learner that has discovered the one-to-one relationship in the synthetic data should have a perfect ME score of 1.0. In Figure 2a, the probability assigned to the novel output symbol is 0.01 and thus the corresponding ME Score is 0.01. The challenge is to get a high ME score for novel (test) items while also correctly classifying known (training) items. A wide range of standard neural networks are evaluated on the mutual exclusivity test. We use an embedding layer to map the input symbols to vectors of size 20 or 100, followed optionally by a hidden layer, and then by a 100-way softmax output layer. The networks are trained with different activation functions (ReLUs , TanH, Sigmoid), optimizers (Adam , Momentum, SGD), learning rates (0.1, 0.01, 0.001) and regularizers (weight decay, batch-normalisation , dropout (b), and entropy regularization (see Appendix B.1)). The models are trained to maximize log-likelihood. All together, we evaluated over 400 different models on the synthetic ME task. Figure 4: Ideal and untrained ME scores compared with the ME scores of a few learned models. Results. Several representative training runs with different architectures are shown in Figure 3. An ideal learner that has discovered the one-to-one pattern should have a mutual exclusivity of 1; for a novel input, the network would assign all the probability mass to the unseen output symbols. In contrast, none of the configurations and architectures tested behave in this way. As training progresses, the mutual exclusivity score (solid blue line; Figure 3) tends to fall along with the training loss (red line). In fact, almost all of the networks acquire a strong anti-mutual exclusivity bias, transitioning from a initial neutral bias to placing most or all of the probability mass on familiar outputs (seen in Figure 4). An exception to this pattern is the entropy regularized model, which maintains a score equivalent to an untrained network. In general, trained models strongly predict that a novel input symbol will correspond to a known rather than unknown output symbol, in contradiction to ME and the organizing structure of the synthetic data. Further informal experiments suggest our cannot be reduced to simply not enough data: these architectures do not learn this one-to-one regularity regardless of how many input/output symbols are provided in the training set. Even with thousands of training examples demonstrating a one-toone pattern, the networks do not learn this abstract principle and fail to capture this defining pattern in the data. Other tweaks were tried in an attempt to induce ME, including eliminating the bias units or normalizing the weights, yet we were unable to find an architecture that reliably demonstrated the ME effect. The show that standard neural networks fail to reason by mutual exclusivity when trained in a variety of typical settings. The models fail to capture the perfect one-to-one mapping (ME bias) seen in the synthetic data, predicting that new symbols map to familiar outputs in a many-to-many fashion. Although our focus is on neural networks, this characteristic is not unique to this model class. We posit it more generally affects flexible models trained to maximize log-likelihood. In a trained network, the optimal activation value for an unused output node is zero: for any given training example, increasing value of an unused output simply reduces the available probability mass for the Name Languages Sentence Pairs Vocabulary Size IWSLT'14 Eng.-Vietnamese ∼133K 17K(en), 7K(vi) WMT'14 Eng.-German ∼4.5M 50K(en), 50K(de) WMT'15 Eng.-Czech ∼15.8M 50K(en), 50K(cs) target output. Using other loss functions could in different outcomes, but we also did not find that weight decay and entropy regularization of reasonable values could fundamentally alter the use of novel outputs. In the next section, we investigate if the lack of ME could hurt performance on common learning tasks such as machine translation and image classification. Mutual exclusivity has implications for a variety of common learning settings. Mutual exclusivity arises naturally in lifelong learning settings, which more realistically reflect the "open world" characteristics of human cognitive development. Unlike epoch-based learning, a lifelong learning agent does not assume a fixed set of concepts and categories. Instead, new concepts can be introduced at any point during learning. An intelligent learner should be sensitive to this possibility, and ME is one means of intelligently reasoning about the meaning of novel stimuli. Children and adults learn in an open world with some probability of encountering a new class at any point, resembling the first epoch of training a neural net only. Moreover, the distribution of categories is neither uniformly distributed nor randomly shuffled . To simulate these characteristics, we construct lifelong learning scenarios using standard benchmarks as described below. In this section, we investigate if mutual exclusivity could be a helpful bias when training machine translation models in a lifelong learning paradigm. From the previous experiments, we know that the type of sequence-to-sequence (seq2seq) models used for translation acquire an anti-ME bias over the course of training (see Appendix A). Would a translation system benefit from assuming that a single word in the source sentence maps to a single word in the target sentence, and vice-versa? This assumption is not always correct since synonymy and polysemy are prevalent in natural languages, and thus the answer to whether or not ME holds is not absolute. Instead, we seek to measure the degree to which this bias holds in lifelong learning on real datasets, and compare this bias to the inductive biases of models trained on these datasets. The data for translation provides a somewhat natural distribution over the frequency at which different words are observed (there are words that appear much more frequently than the others). This allows us to use a single pass through the dataset as a proxy for lifelong translation learning. Datasets. We analyze three common datasets for machine translation, each consisting of pairs of sentences in two languages (Table 1). The vocabularies are truncated based on word frequency in accordance with the standard practices for training neural machine translation models . Mutual exclusivity. There are several ways to operationalize mutual exclusivity in a machine translation setting. Mutual exclusivity could be interpreted as whether a new word in the source sentence ("Xylophone" in English) is likely to be translated to a new word in the target sentence ("Xylophon" in German), as opposed to a familiar word. Since the word alignments are difficult to determine and not provided with the datasets, we instead measure a reasonable proxy: if a new word is encountered in the source sequence, is a new word also encountered in the target sentence? For a source sentence S and an arbitrary novel word N S, and a target sentence T and a novel word N T, we measure a dataset's ME Score as the conditional probability P (N T ∈ T |N S ∈ S). A hypothetical translation model could compute whether or not N S ∈ S by checking if the present word is absent from the vocabulary-so-far during the training process. Thus this conditional probability is an easily-computable cue for determining whether or not a model should expect a novel output word. For the three datasets, we consider both forward and backward translation to get six scenarios for analysis. The probability P (N T ∈ T |N S ∈ S) is estimated for a sample of 100 randomly shuffled sequences of the dataset sentence pairs. See Appendix B.3 for details on calculating the base rate P (N T ∈ T). Table 2: Number of sentences after which the ME Score P (NT ∈ T |NS ∈ S) falls below threshold. Results and Discussion. The measures of conditional probability in the six scenarios are shown in Table 2. There is a consistent pattern through the trajectory of early learning: the conditional probability P (N T ∈ T |N S ∈ S) is high initially for thousands of initial sentence presentations, but then wanes as the network encounters more samples from the dataset. For a large part of the initial training, a seq2seq model would benefit from predicting that previously unseen words in the source language are more likely to map to unseen words in the target language. Moreover, this conditional probability is always higher than the base rate of encountering a new word, indicating that conditioning on the novelty of the input provides substantial additional signal for predicting novelty in the output. Nevertheless, even the base rate suggests that a model should expect novel words with some regularity in our settings. This is in stark contrast to the synthetic showing that seq2seq models quickly acquire an anti-ME assumption (see Appendix A), and their expectation of mapping novel inputs to novel outputs decays rapidly as training progresses (Appendix Figure 7). Similar to translation, we examine if object classifiers would benefit from reasoning by mutual exclusivity during training processes that mirror lifelong learning. To study this, when selecting an image for training, we sample the class from a power law distribution (see Appendix B.2) such that the model is more likely to see certain classes . Ideally, we would model the probability that an object belongs to a novel class based on its similarity to previous samples seen by the model (e.g., outlier detection). Identifying that an image belongs to a novel class is non-trivial, and instead we calculate the base rate for classifying an image as "new" while a learner progresses through the dataset. The set of classes not seen by the model are referred to as "new" here. This measure can be seen as a lower bound on the usefulness of ME through the standard training process, since this calculation assumes a blind learner that is unaware of any novelty signal present in the raw image. Datasets. This section examines the Omniglot dataset and the ImageNet dataset . The Omniglot dataset has been widely used to study few-shot learning, consisting of 1623 classes of handwritten characters with 20 images per class. The ImageNet dataset consists of about 1.2 million images from 1000 different classes. Mutual exclusivity. To measure ME throughout the training process, we examine if an image encountered for the first time while training belongs to a class that has not been seen before. This is operationalized as the probability of encountering an image from a new class N as a function of the number of images seen so far t, P (N |t) (see Appendix B.5). This analysis is agnostic to the content of the image and whether or not it is a repeated item; it only matters whether or not the class is novel. As before, the analysis is performed using ten random runs through the dataset. We contrast the statistics of the datasets by comparing them to the ME Score (Equation 1) of neural network classifiers trained on the datasets. See Appendix B.4 for details about the models and their training procedures. The probability mass assigned to the unseen classes by the network is recorded after each optimizer step, as computed using Equation 1. Results and Discussion. The are summarized in Figure 6 and Table 3. The probability that a new image belongs to an unseen class P (N |t) is higher than the ME score of the classifier through most of the learning phase. Comparing the statistics of the datasets to the inductive biases in the classifiers, the ME score for the classifiers is substantially lower than the baseline ME measure in the dataset, P (N |t) (Table 3). For instance, the ImageNet classifier drops its ME score below 0.05 after about 8,960 images, while the approximate ME measure for the dataset shows that new classes are encountered at above this rate until at least 111,000 images. We found that higher learning rates can force the probabilities assigned to unseen classes to zero on ImageNet after just a single gradient step. These suggest that neural classifiers, with their bias favoring frequent rather than infrequent outputs for novel stimuli, are not well-suited to lifelong learning challenges where such inferences are critical. Although we examined classifiers trained in an online fashion, we would expect similar when we train them using replay or epoch-based training setups, where repeated presentation of past examples would only strengthen the anti-ME bias. These classifiers are hurt by their lack of ME and their failure to consider that new stimuli likely map to new classes. Ideally, a learning algorithm should be capable of leveraging the image content, combined with its own learning maturity, to decide how strongly it should reason by ME. Instead, standard models and training procedures do not provide these capabilities and do not utilize this important inductive bias observed in cognitive development. Children use the mutual exclusivity (ME) bias to learn the meaning of new words efficiently, yet standard neural networks learn very differently. Our show that standard deep learning algorithms lack the ability to reason with ME, including feedforward networks and recurrent sequenceto-sequence models trained to maximize log-likelihood with common regularizers. Beyond simply lacking this bias, these networks learn an anti-ME bias, preferring to map novel inputs to familiar and frequent (rather than unfamiliar) output classes. Our also show that these characteristics The plots show the probability that a new input image belongs to an unseen class P (N |t), as a function of the number of images t seen so far during training (blue), with its standard deviation. This measure is contrasted with the ME score of a neural network classifier trained through a similar run of the dataset (orange). are poorly matched to more realistic lifelong learning scenarios where novel classes can appear at any point, as demonstrated in the translation and classification experiments presented here. Neural nets may be currently stymied by their lack of ME bias, ignoring a powerful assumption about the structure of learning tasks. Mutual exclusivity is relevant elsewhere in machine learning. Recent work has contrasted the ability of humans and neural networks to learn compositional instructions from just one or a few examples, finding that neural networks lack the ability to generalize systematically (; . The authors suggest that people rely on ME in these learning situations, and thus few-shot learning approaches could be improved by utilizing this bias as well. In our analyses, we show that neural networks tend to learn the opposite bias, preferring to map novel inputs to familiar outputs. More generally, ME can be generalized from applying to "novel versus familiar" stimuli to instead handling "rare versus frequent" stimuli (e.g., in translation, rare source words may map to rare target words). The utility of reasoning by ME could be extended to early stages of epoch based learning too. For example, during epoch-based learning, neural networks take longer to acquire rare stimuli and patterns of exceptions , often mishandling these items for many epochs by mapping them to familiar responses. Another direction for future work is studying how the ME bias should interact with hierarchical categorization tasks. We posit that the ME assumption will be increasingly important as learners tackle more continual, lifelong, and large-scale learning challenges . Mutual exclusivity is an open challenge for deep neural networks, but there are promising avenues for progress. The ME bias will not be helpful for every problem, but it is equally clear that the status quo is sub-optimal: models should not have a strong anti-ME bias regardless of the task and dataset demands. Ideally, a model would decide autonomously how strongly to use ME (or not) based on the demands of the task. For instance, in our synthetic example, an ideal learner would discover the one-to-one correspondence and use this perfect ME bias as a meta-strategy. If the dataset has more many-to-one correspondences, it would adopt another meta-strategy. This meta-strategy could even change depending on the stage of learning, yet such an approach is not currently available for training models. Previous cognitive models of word learning have found ways to incorporate the ME bias (; ; ;), although in ways that do not generalize to training deep neural networks. While successful in some domains, these models are highly simplified or require built-in mechanisms for implementing ME, making them so far impractical for use in realistic settings. As outlined above, it would be ideal to acquire a ME bias via meta learning or learning to learn , with the advantage of calibrating the bias to the dataset itself rather than assuming its strength a priori. For example, the meta learning model of seems capable of learning an ME bias, although it was not specifically probed in this way. Recent work by demonstrated that neural nets can learn to reason by ME if trained explicitly to do so, showing these abilities are within the repertoire of modern tools. However acquiring ME is just one step toward the goal proposed here: using ME to facilitate efficient lifelong learning or large-scale classification and translation. In , standard deep neural networks do not naturally reason by mutual exclusivity, but designing them to do so could lead to faster and more flexible learners. There is a compelling case for building models that learn through mutual exclusivity. Synthetic data. We evaluate if a different class of models, sequence-to-sequence (seq2seq) neural networks , take better advantage of ME structure in the data. This popular class of models is used in machine translation and other natural language processing tasks, and thus the nature of their inductive biases is critical for many applications. As in the previous section, we create a synthetic dataset that has a perfect ME bias: each symbol in the source maps to exactly one symbol in the target. We also have a perfect alignment in the dataset, so that each token in the source corresponds to the token at the same position in the target. The task is illustrated in Figure 2b. We consider translation from sequences of words to sequences of referent symbols. The dataset consists of 20 label-referent pairings. Ten pairs are used to train the model and familiarize the learning algorithm with the task. The remaining ten pairs were used in the test phase. To train the model, we generate 1000 sequences of words whose lengths range from 1 to 5. To generate sequences for the test phase, words in the training sequences are replaced with new words with a probability of 0.2. Thus, 1000 sequences are used to test for ME. The ME score is evaluated using Equation 1 at positions in the output sequence where the corresponding input is new. As shown in Figure 2b, the ME score is evaluated in the second position which corresponds to the novel word "dax," using the probability mass assigned to the unseen output symbols. We probe a seq2seq model that has a recurrent encoder using Gated Recurrent Units (GRUs) and a GRU decoder. Both the encoder and decoder had embedding and hidden sizes of 256. Dropout (a) with a probability of 0.5 was used during training. The networks are trained using an Adam optimizer with a learning rate of 0.001 and a log-likelihood loss. Two versions of the network were trained with and without attention. Results. Results are shown in Figure 7 and confirm our findings from the feedforward network. The ME score falls to zero within a few training iterations, and the networks fail to assign any substantial probability to the unseen classes. The networks achieve a perfect score on the training set, but cannot extrapolate the one-to-one mappings to unseen symbols. Again, not only do seq2seq models fail to show a mutual exclusivity bias, they acquire a anti-mutual exclusivity bias that goes against the structure of the dataset. B ADDITIONAL DETAILS The entropy regularizer is operationalized by subtracting the entropy of prediction from the loss. Thus, the model is penalized for being overly confident about its prediction. We found that the entropy regularizer produces an ME score that stays constant across training, at the cost of the model being less confident about predictions made for seen classes. To simulate a naturalistic lifelong learning scenario, we try to sample a few classes more frequently than the others. To do this, we sample the classes for training using a power law distribution. Weights are assigned to the classes using the following formula, where c is the index of the class. After the class for training is chosen, we uniformly sample from the images of that class for training. The base rate for machine translation is defined as the probability of observing a new word in the target at a particular time t in training. We go through the unseen sentences in the corpus from the target language at time t to compute the probability of sampling a sentence with at least one new word. Thus, the base rate at time t in training is defined as: P (new in target at t) = # of unseen sentences in target with new words # of unseen sentences For Omniglot, a convolutional neural network was trained on 1623-way classification. The architecture consists of 3 convolutional layers (each consisting of 5 × 5 kernels and 64 feature maps), a fully connected layer (576 × 128) and a softmax classification layer. It was trained with a batch size of 16 using an Adam optimizer and a learning rate of 0.001. For Imagenet, a Resnet18 model was trained on 1000-way classification with a batch size of 256, using an Adam optimizer and a learning rate of 0.001. For classification, we calculate the score P (N |t) for the model by adding the probabilities the model assigns to all the "new" (unseen) classes when iterating through the remaining corpus (similar to Equation 1). For the dataset, we compute P (N |t) by sampling all unseen images in the corpus and compute the proportion from "new" classes given their ground truth labels.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Children use the mutual exclusivity (ME) bias to learn new words, while standard neural nets show the opposite bias, hindering learning in naturalistic scenarios such as lifelong learning.

Represent the TLDR for finding the corresponding scientific article

We can identify prototypical and outlier examples in machine learning that are quantifiably very different, and make use of them to improve many aspects of neural networks.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Cortical neurons process and integrate information on multiple timescales. In addition, these timescales or temporal receptive fields display functional and hierarchical organization. For instance, areas important for working memory (WM), such as prefrontal cortex, utilize neurons with stable temporal receptive fields and long timescales to support reliable representations of stimuli. Despite of the recent advances in experimental techniques, the underlying mechanisms for the emergence of neuronal timescales long enough to support WM are unclear and challenging to investigate experimentally. Here, we demonstrate that spiking recurrent neural networks (RNNs) designed to perform a WM task reproduce previously observed experimental findings and that these models could be utilized in the future to study how neuronal timescales specific to WM emerge. Previous studies have shown that higher cortical areas such as prefrontal cortex operate on a long timescale, measured as the spike-count autocorrelation decay constant at rest. These long timescales have been hypothesized to be critical for performing working memory (WM) computations, but it is experimentally challenging to probe the underlying circuit mechanisms that lead to stable temporal properties. Recurrent neural network (RNN) models trained to perform WM tasks could be a useful tool if these models also utilize units with long heterogeneous timescales and capture previous experimental findings. However, such RNN models have not yet been identified. In this study, we construct a spiking RNN model to perform a WM task and compare the emerging timescales with the timescales derived from the prefrontal cortex of rhesus monkeys trained to perform similar WM tasks. We show that both macaque prefrontal cortex and the RNN model utilize units/neurons with long timescales during delay period to sustain stimulus information. In addition, the number of units with long timescales was significantly reduced in the RNN model trained to perform a non-WM task, further supporting the idea that neuronal timescales are task-specific and functionally organized. We employed a spiking RNN model based on leaky integrate-and-fire (LIF) units recurrently connected to one another. These units are governed by: 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada. where τ m is the membrane time constant (10 ms), v i (t) is the membrane voltage of unit i at time t, x i (t) is the synaptic input current that unit i receives at time t, I ext is the external input current, and R is the leak resistance (set to 1). The synaptic input current (x) is modeled using a single-exponential synaptic filter applied to the presynaptic spike trains: where τ s i is the synaptic decay time constant of unit i, w ij defines the synaptic connectivity strength from unit j to unit i, and the second summation refers to the spike train produced by unit j. We used the method that we previously developed in to construct LIF RNNs that performed a delayed match-to-sample task (DMS; Figure 1A top). Briefly, we trained several continuous-variable rate RNNs to perform the DMS task using a gradient descent algorithm, and the trained networks were then mapped to LIF networks. In total, we trained 40 RNNs of 200 units (80% excitatory and 20% inhibitory units) to perform the task. The synaptic decay constants (τ s) were optimized and constrained to vary between 20 ms and 125 ms, but the major findings presented here did not change when the synaptic decay constants were not optimized (i.e. fixed to a constant value; see Section 4). All the units from the trained RNNs that satisfied the preprocessing criteria were pooled for the spike-count autocorrelation analysis (see Section 4). The task began with a 1 s of fixation period (i.e. no external input) followed by two sequential input stimuli (each stimulus lasting for 0.25 s) separated by a delay period (0.75 s). The input signal was set to either -1 or +1 during the stimulus window. If the two sequential stimuli had the same sign (-1/-1 or +1/+1), the network was trained to produce an output signal approaching +1 after the offset of the second stimulus (Figure 1A). If the stimuli had opposite signs (-1/+1 or +1/-1), the network produced an output signal approaching -1 (Figure 1A). To ensure that the spiking RNN model we employed here is a valid model for investigating neuronal timescales observed in the prefrontal cortex, we compared the findings from our model to the findings obtained from a publicly available dataset. The dataset contains single-neuron spike train recordings from ventral and dorsal prefrontal cortex of four rhesus monkeys performing DMS tasks (Figure 1B). The spike trains of 3257 neurons recorded in the dorsolateral prefrontal cortex (dlPFC) were analyzed for the spike-count autocorrelation analysis. More details regarding the dataset and the tasks can be found in. 4 Neuronal timescales specific to working memory Spike-count autocorrelation decay time constants. To characterize the temporal receptive field, we computed the decay time constant of the spike-count autocorrelation for each unit/neuron during the fixation period. For each unit/neuron, we first binned the spike trains (during the fixation period) over multiple trials using a non-overlapping 50-ms moving window. Since the fixation period duration was 1 s for both RNN and experimental models, this ed in a [Number of Trials × 20] spike-count matrix for each unit/neuron. For the experimental data, the minimum number of trials required for a neuron to be considered for analysis was 11 trials. The average number of trials from all the neurons included in the analysis was 84.8 ± 34.5 trials. For the RNN model, we generated 50 trials for each unit. Next, a Pearson's correlation coefficient (ρ) was computed between two time bins (i.e. two columns in the spike-count matrix) separated by a lag (∆). The coefficient was calculated for all possible pairs with the maximum lag of 650 ms. The coefficients were averaged for each lag value, and an exponential decay function was fitted across the average coefficient values (ρ) using the LevenbergMarquardt nonlinear least-squares method: where A and B are the amplitude and the offset of the fit, respectively. The timescale (σ) defines how fast the autocorrelation decays. The following three inclusion criteria (commonly used in previous experimental studies) were applied to the RNN model and the experimental data: 0 < σ ≤ 500 ms, A > 0, and a first decrease in ρ earlier than ∆ = 150 ms. In addition, the fitting was started after a first decrease in autocorrelation. As shown in Figure 2 (left), the timescales extracted from the untrained RNNs (sparse, random Gaussian connectivity weights; 2769 units from 40 RNNs satisfied the inclusion criteria) were right-skewed. On the other hand, the trained RNNs (841 units from 40 RNNs satisfied the inclusion criteria) and the experimental data (959 units from 4 monkeys satisfied the inclusion criteria) were heavily left-skewed, suggesting that both trained model and data contained predominantly units with long timescales (Figure 2). The distributions and the average autocorrelation values from the RNN model and the experimental data were within those previously reported. Units with long neuronal timescales encode information robustly. Next, we investigated to see if the units/neurons with longer timescales were involved with more stable coding using cross-temporal decoding analysis. For each cue stimulus identity, the trials of each neuron were divided into two splits in an interleaved manner (i.e. even vs odd trials). All possible pairwise differences (in instantaneous firing rates) between cue conditions were computed within each split. Finally, a Fisher-transformed Pearson correlation coefficient was computed between the pairwise differences of the first split at time t 1 and the differences of the second split at time t 2. A high Fisher-transformed correlation value (i.e. high discriminability) represents a reliable cue-specific difference present in the network population. We performed the above analysis on short and long neuronal timescale subgroups from the experimental data and the RNN model. A unit/neuron was assigned to the short σ group if its timescale was smaller than the lower quartile value. The upper quartile was used to identify units/neurons with large autocorrelation decay time constants. There were 122 short σ and 128 long σ neurons for the experimental data. For the RNN model, there were 210 units in each subgroup. The cross-temporal discriminability matrices (Figure 3) indicate that stronger cue-specific differences across the delay period were present in the long σ subgroup compared to the short σ subgroup for both experimental data and the RNN model. These are consistent with the previous experimental findings, and suggest that longer neuronal timescales correspond to more stable coding. Task-specific temporal receptive fields. Murray et al. demonstrated that the hierarchical organization of the neuronal timescales from different cortical areas closely tracks the anatomical hierarchical organization. For instance, sensory areas important for detecting incoming stimuli house predominantly neurons with short timescales. On the other hand, higher cortical areas including prefrontal areas may require neurons with stable temporal receptive fields that are capable of encoding and integrating information on a longer timescale. Timescale (ms) To investigate if such functional specialization also emerges in our spiking model, we trained another group of spiking RNNs (40 RNNs) on a simpler task that did not require WM. The task paradigm is modeled after a Go-NoGo task, and required the RNNs to respond immediately after the cue stimulus: output approaching -1 for the "-1" cue and +1 for the "+1" cue. Each cue stimulus lasted for 125 ms. This specific task paradigm, which we refer to as Go-NoGo task, was chosen since it does not involve WM, and it has been widely used to study how primary sensory areas process sensory information. Apart from the task paradigm, all the other model parameters were same as the RNNs trained to perform the DMS task. The autocorrelation decay timescales extracted from the RNNs trained to perform the Go-NoGo task were significantly shorter than the timescales obtained from the working memory RNNs (Figure 4). The RNNs contained fewer units with long timescales compared to the DMS RNNs (Figure 4A), and the timescales averaged by network were also significantly faster than the average timescales from the DMS networks (Figure 4B). In addition, the average autocorrelation timecourse for the Go-NoGo networks decayed faster than the one from the DMS RNNs and resembled the timecourses obtained from the primary somatosensory cortex and the medial-temporal area in the visual cortex (see Figure 1C in ). These findings indicate that the neuronal timescales of our RNN models are task-specific and possibly organized in a hierarchical fashion. Timescale (ms) Effects of synaptic decay constants on neuronal timescales. In this section, we demonstrate that the timescales that we obtained from the two RNN models (Go-NoGo and DMS) in the previous section are not largely driven by the synaptic decay time constants (τ s) that we optimized to construct the spiking RNNs. For each task model, we trained 40 RNNs with the synaptic decay constants fixed to 125 ms. Even though all the units now had τ s = 125 ms, the timescale distributions from both models were largely preserved (Figure 5A), and the hierarchy was also maintained (Figure 5B). For the Go-NoGo model, the average timescale values (averaged by network) obtained from the τ s optimized RNNs were significantly smaller than the timescales computed from the RNNs with τ s = 125 ms: 93.8 ± 27.6 ms and 109.1 ± 21.0 ms for the τ s optimized and fixed RNNs, respectively. On the other hand, the average timescales from the τ s optimized DMS RNNs were significantly larger than the ones extracted from the fixed τ s networks: 157.0 ± 32.0 ms and 141.1 ± 27.6 ms for the τ s optimized and fixed RNNs, respectively. Therefore, increasing the synaptic decay time constant for all the units to 125 ms did not necessarily lead to increased neuronal timescales, suggesting that connectivity patterns and structures might play a bigger role in governing task-specific timescales. In this study, we employed a spiking RNN model of WM to investigate if the model exhibits and utilizes heterogeneous timescales for prolonged integration of information. We validated the model using an experimental dataset obtained from rhesus monkeys trained on WM tasks: the model and the primate prefrontal cortex both displayed similar heterogeneous neuronal timescales and incorporated units/neurons with long timescales to maintain stimulus information. The timescales from the RNN model trained on a non-WM task (Go-NoGo task) were markedly shorter, since units with long timescales were not required to support the simple computation. Future works include characterizing the network dynamics and the circuit motifs of the DMS RNN model to elucidate connectivity structures required to give rise to the diverse, stable temporal receptive fields specific to WM.

Spiking recurrent neural networks performing a working memory task utilize long heterogeneous timescales, strikingly similar to those observed in prefrontal cortex.

Represent the short summary for finding the corresponding scientific article (long)

We detect statistical interactions captured by a feedforward multilayer neural network by directly interpreting its learned weights.