Question: Represent the article for finding a single-sentence TLDR

Question: Disentangled representations, where the higher level data generative factors are reflected in disjoint latent dimensions, offer several benefits such as ease of deriving invariant representations, transferability to other tasks, interpretability, etc. We consider the problem of unsupervised learning of disentangled representations from large pool of unlabeled observations, and propose a variational inference based approach to infer disentangled latent factors. We introduce a regularizer on the expectation of the approximate posterior over observed data that encourages the disentanglement. We also propose a new disentanglement metric which is better aligned with the qualitative disentanglement observed in the decoder's output. We empirically observe significant improvement over existing methods in terms of both disentanglement and data likelihood (reconstruction quality). Feature representations of the observed raw data play a crucial role in the success of machine learning algorithms. Effective representations should be able to capture the underlying (abstract or high-level) latent generative factors that are relevant for the end task while ignoring the inconsequential or nuisance factors. Disentangled feature representations have the property that the generative factors are revealed in disjoint subsets of the feature dimensions, such that a change in a single generative factor causes a highly sparse change in the representation. Disentangled representations offer several advantages -(i) Invariance: it is easier to derive representations that are invariant to nuisance factors by simply marginalizing over the corresponding dimensions, (ii) Transferability: they are arguably more suitable for transfer learning as most of the key underlying generative factors appear segregated along feature dimensions, (iii) Interpretability: a human expert may be able to assign meanings to the dimensions, (iv) Conditioning and intervention: they allow for interpretable conditioning and/or intervention over a subset of the latents and observe the effects on other nodes in the graph. Indeed, the importance of learning disentangled representations has been argued in several recent works BID5 BID37 BID50.Recognizing the significance of disentangled representations, several attempts have been made in this direction in the past BID50. Much of the earlier work assumes some sort of supervision in terms of: (i) partial or full access to the generative factors per instance BID48 BID58 BID35 BID33, (ii) knowledge about the nature of generative factors (e.g, translation, rotation, etc.) BID29 BID11, (iii) knowledge about the changes in the generative factors across observations (e.g., sparse changes in consecutive frames of a Video) BID25 BID57 BID21 BID14 BID32, (iv) knowledge of a complementary signal to infer representations that are conditionally independent of it 1 BID10 BID41 BID53. However, in most real scenarios, we only have access to raw observations without any supervision about the generative factors. It is a challenging problem and many of the earlier attempts have not been able to scale well for realistic settings BID51 BID15 BID13 ) (see also,).Recently, BID9 proposed an approach to learn a generative model with disentangled factors based on Generative Adversarial Networks (GAN) BID24, however implicit generative models like GANs lack an effective inference mechanism 2, which hinders its applicability to the problem of learning disentangled representations. More recently, proposed an approach based on Variational AutoEncoder (VAE) BID34 for inferring disentangled factors. The inferred latents using their method (termed as β-VAE) are empirically shown to have better disentangling properties, however the method deviates from the basic principles of variational inference, creating increased tension between observed data likelihood and disentanglement. This in turn leads to poor quality of generated samples as observed in.In this work, we propose a principled approach for inference of disentangled latent factors based on the popular and scalable framework of amortized variational inference BID34 BID55 BID23 BID49 powered by stochastic optimization BID30 BID34 BID49. Disentanglement is encouraged by introducing a regularizer over the induced inferred prior. Unlike β-VAE, our approach does not introduce any extra conflict between disentanglement of the latents and the observed data likelihood, which is reflected in the overall quality of the generated samples that matches the VAE and is much better than β-VAE. This does not come at the cost of higher entanglement and our approach also outperforms β-VAE in disentangling the latents as measured by various quantitative metrics. We also propose a new disentanglement metric, called Separated Attribute Predictability or SAP, which is better aligned with the qualitative disentanglement observed in the decoder's output compared to the existing metrics. We start with a generative model of the observed data that first samples a latent variable z ∼ p(z), and an observation is generated by sampling from p θ (x|z). The joint density of latents and observations is denoted as p θ (x, z) = p(z)p θ (x|z). The problem of inference is to compute the posterior of the latents conditioned on the observations, i.e., p θ (z|x) = DISPLAYFORM0. We assume that we are given a finite set of samples (observations) from the true data distribution p(x). In most practical scenarios involving high dimensional and complex data, this computation is intractable and calls for approximate inference. Variational inference takes an optimization based approach to this, positing a family D of approximate densities over the latents and reducing the approximate inference problem to finding a member density that minimizes the Kullback-Leibler divergence to the true posterior, i.e., q * x = min q∈D KL(q(z) p θ (z|x)) BID6. The idea of amortized inference BID34 BID55 BID23 BID49 is to explicitly share information across inferences made for each observation. One successful way of achieving this for variational inference is to have a so-called recognition model, parameterized by φ, that encodes an inverse map from the observations to the approximate posteriors (also referred as variational autoencoder or VAE) BID34 BID49. The recognition model parameters are learned by optimizing the problem min φ E x KL(q φ (z|x) p θ (z|x)), where the outer expectation is over the true data distribution p(x) which we have samples from. This can be shown as equivalent to maximizing what is termed as evidence lower bound (ELBO): DISPLAYFORM1 The ELBO (the objective at the right side of Eq. 1) lower bounds the log-likelihood of observed data, and the gap vanishes at the global optimum. Often, the density forms of p(z) and q φ (z|x) are chosen such that their KL-divergence can be written analytically in a closed-form expression (e.g., p(z) is N (0, I) and q φ (z|x) is N (µ φ (x), Σ φ (x))) BID34. In such cases, the ELBO can be efficiently optimized (to a stationary point) using stochastic first order methods where both expectations are estimated using mini-batches. Further, in cases when q φ (·) can be written as a continuous transformation of a fixed base distribution (e.g., the standard normal distribution), a low variance estimate of the gradient over φ can be obtained by coordinate transformation (also referred as reparametrization) BID22 BID34 BID49. Most VAE based generative models for real datasets (e.g., text, images, etc.) already work with a relatively simple and disentangled prior p(z) having no interaction among the latent dimensions (e.g., the standard Gaussian N (0, I)) BID7 BID43 BID31 BID59. The complexity of the observed data is absorbed in the conditional distribution p θ (x|z) which encodes the interactions among the latents. Hence, as far as the generative modeling is concerned, disentangled prior sets us in the right direction. Although the generative model starts with a disentangled prior, our main objective is to infer disentangled latents which are potentially conducive for various goals mentioned in Sec. 1 (e.g., invariance, transferability, interpretability). To this end, we consider the density over the inferred latents induced by the approximate posterior inference mechanism, DISPLAYFORM0 which we will subsequently refer to as the inferred prior or expected variational posterior (p(x) is the true data distribution that we have only samples from). For inferring disentangled factors, this should be factorizable along the dimensions, i.e., DISPLAYFORM1 This can be achieved by minimizing a suitable distance between the inferred prior q φ (z) and the disentangled generative prior p(z). We can also define expected posterior as DISPLAYFORM2 If we take KL-divergence as our choice of distance, by relying on its pairwise convexity (i.e., KL( BID56, we can show that the distance between q φ (z) and p θ (z) is bounded by the objective of the variational inference: DISPLAYFORM3 DISPLAYFORM4 In general, the prior p(z) and expected posterior p θ (z) will be different, although they may be close (they will be same when p θ (x) = p θ (x|z)p(z)dz is equal to p(x)). Hence, variational posterior inference of latent variables with disentangled prior naturally encourages inferring factors that are close to being disentangled. We think this is the reason that the original VAE (Eq. ) has also been observed to exhibit some disentangling behavior on simple datasets such as MNIST BID34. However, this behavior does not carry over to more complex datasets BID4 BID39, unless extra supervision on the generative factors is provided BID35 BID33. This can be due to: (i) p(x) and p θ (x) being far apart which in turn causes p(z) and p θ (z) being far apart, and (ii) the non-convexity of the ELBO objective which prevents us from achieving the global minimum of E x KL(q φ (z|x) p θ (z|x)) (which is 0 and implies KL(q φ (z) p θ (z)) = 0). In other words, maximizing the ELBO (Eq.) might also in reducing the value of KL(q φ (z) p(z)), however, due to the aforementioned reasons, the gap between KL(q φ (z) p(z)) and E x KL(q φ (z|x) p θ (z|x)) could be large at the stationary point of convergence. Hence, minimizing KL(q φ (z) p(z)) or any other suitable distance D(q φ (z), p(z)) explicitly will give us better control on the disentanglement. This motivates us to add D(q φ (z) p(z)) as part of the objective to encourage disentanglement during inference, i.e., DISPLAYFORM5 where λ controls its contribution to the overall objective. We refer to this as DIP-VAE (for Disentangled Inferred Prior) subsequently. Optimizing FORMULA7 directly is not tractable if D(·, ·) is taken to be the KL-divergence KL(q φ (z) p(z)), which does not have a closed-form expression. One possibility is use the variational formulation of the KL-divergence BID45 BID46 ) that needs only samples from q φ (z) and p(z) to estimate a lower bound to KL(q φ (z) p(z)). However, this would involve optimizing for a third set of parameters ψ for the KL-divergence estimator, and would also change the optimization to a saddle-point (min-max) problem which has its own optimization challenges (e.g., gradient vanishing as encountered in training generative adversarial networks with KL or Jensen-Shannon (JS) divergences BID24 BID2 ). Taking D to be another suitable distance between q φ (z) and p(z) (e.g., integral probability metrics like Wasserstein distance BID54) might alleviate some of these issues but will still involve complicating the optimization to a saddle point problem in three set of parameters 3. It should also be noted that using these variational forms of the distances will still leave us with an approximation to the actual distance. We adopt a simpler yet effective alternative of matching the moments of the two distributions. Matching the covariance of the two distributions will amount to decorrelating the dimensions of DISPLAYFORM6 By the law of total covariance, the covariance of z ∼ q φ (z) is given by DISPLAYFORM7 where E q φ (z|x) [z] and Cov q φ (z|x) [z] are random variables that are functions of the random variable x (z is marginalized over). Most existing work on the VAE models uses q φ (z|x) having the form DISPLAYFORM8, where µ φ (x) and Σ φ (x) are the outputs of a deep neural net parameterized by φ. In this case Eq. reduces to Cov DISPLAYFORM9, which we want to be close to the Identity matrix. For simplicity, we choose entry-wise squared 2 -norm as the measure of proximity. Further, Σ φ (x) is commonly taken to be a diagonal matrix which means that cross-correlations (off-diagonals) between the latents are due to only DISPLAYFORM10. This suggests two possible options for the disentangling regularizer: DISPLAYFORM11 which we refer as DIP-VAE-II. Penalizing just the off-diagonals in both cases will lead to lowering the diagonal entries of Cov p(x) [µ φ (x)] as the ij'th off-diagonal is really a derived attribute obtained by multiplying the square-roots of i'th and j'th diagonals (for each example x ∼ p(x), followed by averaging over all examples). This can be compensated in DIP-VAE-I by a regularizer on the diagonal entries of Cov p(x) [µ φ (x)] which pulls these towards 1. We opt for two separate hyperparameters controlling the relative importance of the loss on the diagonal and off-diagonal entries as follows: DISPLAYFORM12 The regularization terms involving Cov p(x) [µ φ (x)] in the above objective can be efficiently optimized using SGD, where Cov p(x) [µ φ (x)] can be estimated using the current minibatch 4.For DIP-VAE-II, we have the following optimization problem: DISPLAYFORM13 As discussed earlier, the term DISPLAYFORM14 Penalizing the off-diagonals of Cov p(x) [µ φ (x)] in the Objective will contribute to reduction in the magnitude of its diagonals as discussed earlier. As the regularizer on the diagonals is not directly on DISPLAYFORM15, unlike DIP-VAE-I, it will be not be able to keep DISPLAYFORM16 ii such that their sum remains close to 1. In datasets where the number of generative factors is less than the latent dimension, DIP-VAE-II is more suitable than DIP-VAE-I as keeping all dimensions active might in splitting of an attribute across multiple dimensions, hurting the goal of disentanglement. It is also possible to match higher order central moments of q φ (z) and the prior p(z). In particular, third order central moments (and moments) of the zero mean Gaussian prior are zero, hence 2 norm of third order central moments of q φ (z) can be penalized. Recently proposed β-VAE proposes to modify the ELBO by upweighting the KL(q φ (z|x) p(z)) term in order to encourage the inference of disentangled factors: DISPLAYFORM0 where β is taken to be great than 1. Higher β is argued to encourage disentanglement at the cost of reconstruction error (the likelihood term in the ELBO). Authors report empirical with β ranging from 4 to 250 depending on the dataset. As already mentioned, most VAE models proposed in the literature, including β-VAE, work with N (0, I) as the prior p(z) and N (µ φ (x), Σ φ (x)) with diagonal Σ φ (x) as the approximate posterior q φ (z|x). This reduces the objective to DISPLAYFORM1 For high values of β, β-VAE would try to pull µ φ (x) towards zero and Σ φ (x) towards the identity matrix (as the minimum of x − ln x for x > 0 is at x = 1), thus making the approximate posterior q φ (z|x) insensitive to the observations. This is also reflected in the quality of the reconstructed samples which is worse than VAE (β = 1), particularly for high values of β. Our proposed method does not have such increased tension between the likelihood term and the disentanglement objective, and the sample quality with our method is on par with the VAE.Finally, we note that both β-VAE and our proposed method encourage disentanglement of inferred factors by pulling Cov q φ (z) (z) in Eq. towards the identity matrix: β-VAE attempts to do it by making Cov q φ (z|x) (z) close to I and E q φ (z|x) (z) close to 0 individually for all observations x, while the proposed method directly works on Cov q φ (z) (z) (marginalizing over the observations x) which retains the sensitivity of q φ (z|x) to the conditioned-upon observation.3 QUANTIFYING DISENTANGLEMENT: SAP SCORE propose a metric to evaluate the disentanglement performance of the inference mechanism, assuming that the ground truth generative factors are available. It works by first sampling a generative factor y, followed by sampling L pairs of examples such that for each pair, the sampled generative factor takes the same value. Given the inferred z x:= µ φ (x) for each example x, they compute the absolute difference of these vectors for each pair, followed by averaging these difference vectors. This average difference vector is assigned the label of y. By sampling n such minibatches of L pairs, we get n such averaged difference vectors for the factor y. This process is repeated for all generative factors. A low capacity multiclass classifier is then trained on these vectors to predict the identities of the corresponding generative factors. Accuracy of this classifier on the difference vectors for test set is taken to be a measure of disentanglement. We evaluate the proposed method on this metric and refer to this as Z-diff score subsequently. We observe in our experiments that the Z-diff score is not correlated well with the qualitative disentanglement at the decoder's output as seen in the latent traversal plots (obtained by varying only one latent while keeping the other latents fixed). It also depends on the multiclass classifier used to obtain the score. We propose a new metric, referred as Separated Attribute Predictability (SAP) score, that is better aligned with the qualitative disentanglement observed in the latent traversals and also does not involve training any classifier. It is computed as follows: (i) We first construct a d × k score matrix S (for d latents and k generative factors) whose ij'th entry is the linear regression or classification score (depending on the generative factor type) of predicting j'th factor using only i'th latent [µ φ (x)] i. For regression, we take this to be the R 2 score obtained with fitting a line (slope and intercept) that minimizes the linear regression error (for the test examples). The R 2 score is given by DISPLAYFORM2 and ranges from 0 to 1, with a score of 1 indicating that a linear function of the i'th inferred latent explains all variability in the j'th generative factor. For classification, we fit one or more thresholds (real numbers) directly on i'th inferred latents for the test examples that minimize the balanced classification errors, and take S i j to be the balanced classification accuracy of the j'th generative factor. For inactive latent Table 1: Z-diff score, the proposed SAP score and reconstruction error (per pixel) on the test sets for 2D Shapes and CelebA (β 1 = 4, β 2 = 60, λ = 10, λ 1 = 5, λ 2 = 500 for 2D Shapes; β 1 = 4, β 2 = 32, λ = 2, λ 1 = 1, λ 2 = 80 for CelebA). For the on a wider range of hyperparameter values, refer to Fig. 1 DISPLAYFORM3 ] ii close to 0), we take S ij to be 0.(ii) For each column of the score matrix S which corresponds to a generative factor, we take the difference of top two entries (corresponding to top two most predictive latent dimensions), and then take the mean of these differences as the final SAP score. Considering just the top scoring latent dimension for each generative factor is not enough as it does not rule out the possibility of the factor being captured by other latents. A high SAP score indicates that each generative factor is primarily captured in only one latent dimension. Note that a high SAP score does not rule out one latent dimension capturing two or more generative factors well, however in many cases this would be due to the generative factors themselves being correlated with each other, which can be verified empirically using ground truth values of the generative factors (when available). Further, a low SAP score does not rule out good disentanglement in cases when two (or more) latent dimensions might be correlated strongly with the same generative factor and poorly with other generative factors. The generated examples using single latent traversals may not be realistic for such models, and DIP-VAE discourages this from happening by enforcing decorrelation of the latents. However, the SAP score computation can be adapted to such cases by grouping the latent dimensions based on correlations and getting the score matrix at group level, which can be fed as input to the second step to get the final SAP score. We evaluate our proposed method, DIP-VAE, on three datasets -(i) CelebA BID39: It consists of 202, 599 RGB face images of celebrities. We use 64 × 64 × 3 cropped images as used in several earlier works, using 90% for training and 10% for test. (ii) 3D Chairs BID4: It consists of 1393 chair CAD models, with each model rendered from 31 azimuth angles and 2 elevation angles. Following earlier work BID58 BID18 that ignores near-duplicates, we use a subset of 809 chair models in our experiments. We use the binary masks of the chairs as the observed data in our experiments following. First 80% of the models are used for training and the rest are used for test. (iii) 2D Shapes: This is a synthetic dataset of binary 2D shapes generated from the Cartesian product of the shape (heart, oval and square), x-position (32 values), y-position (32 values), scale (6 values) and rotation (40 values). We consider two baselines for the task of unsupervised inference of disentangled factors: (i) VAE BID34 BID49, and (ii) the recently proposed β-VAE. To be consistent with the evaluations in, we use the same CNN network architectures (for our encoder and decoder), and same latent dimensions as used in for CelebA, 3D Chairs, 2D Shapes datasets. Hyperparameters. For the proposed DIP-VAE-I, in all our experiments we vary λ od in the set {1, 2, 5, 10, 20, 50, 100, 500} while fixing λ d = 10λ od for 2D Shapes and 3D Chairs, and λ d = 50λ od for CelebA. For DIP-VAE-II, we fix λ od = λ d for 2D Shapes, and λ od = 2λ d for CelebA. Additionally, for DIP-VAE-II we also penalize the 2 -norm of third order central moments of q φ (z) with hyperparameter λ 3 = 200 for 2D Shapes data (λ 3 = 0 for CelebA). For β-VAE, we experiment with β = {1, 2, 4, 8, 16, 25, 32, 64, 100, 128, 200, 256} (where β = 1 corresponds to the VAE). We used a batch size of 400 for all 2D Shapes experiments and 100 for all CelebA experiments. For both CelebA and 2D Shapes, we show the in terms of the Z-diff score, the Figure 1: Proposed Separated Atomic Predictability (SAP) score and the Z-diff disentanglement score as a function of average reconstruction error (per pixel) on the test set of 2D Shapes data for β-VAE and the proposed DIP-VAE. The plots are generated by varying β for β-VAE, and λ od for DIP-VAE-I and DIP-VAE-II (the number next to each point is the value of these hyperparameters, respectively). DISPLAYFORM0 is computed for every attribute k using the training set and a bias is learned by minimizing the hinge loss. Accuracy on other attributes stays about same across all methods. Arched proposed SAP score, and reconstruction error. For 3D Chairs data, only two ground truth generative factors are available and the quantitative scores for these are saturated near the peak values, hence we show only the latent traversal plots which we based on our subjective evaluation of the reconstruction quality and disentanglement (shown in Appendix).Disentanglement scores and reconstruction error. For the Z-diff score, in all our experiments we use a one-vs-rest linear SVM with weight on the hinge loss C set to 0.01 and weight on the regularizer set to 1. Table 1 shows the Z-diff scores and the proposed SAP Figure 2: The proposed SAP score and the Z-diff score as a function of average reconstruction error (per pixel) on the test set of CelebA data for β-VAE and the proposed DIP-VAE. The plots are generated by varying β for β-VAE, and λ od for DIP-VAE-I and DIP-VAE-II (the number next to each point is the value of these hyperparameters, respectively).scores along with reconstruction error (which directly corresponds to the data likelihood) for the test sets of CelebA and 2D Shapes data. Further we also show the plots of how the Z-diff score and the proposed SAP score change with the reconstruction error as we vary the hyperparameter for both methods (β and λ od, respectively) in Fig. 1 (for 2D Shapes data) and Fig. 2 (for CelebA data). The proposed DIP-VAE-I gives much higher Z-diff score at little to no cost on the reconstruction error when compared with VAE (β = 1) and β-VAE, for both 2D Shapes and CelebA datasets. However, we observe in the decoder's output for single latent traversals (varying a single latent while keeping others fixed, shown in FIG0 and FIG1) that a high Z-diff score is not necessarily a good indicator of disentanglement. Indeed, for 2D Shapes data, DIP-VAE-I has a higher Z-diff score (98.7) and almost an order of magnitude lower reconstruction error than β-VAE for β = 60, however comparing the latent traversals of β-VAE in FIG0 and DIP-VAE-I in FIG1 indicate a better disentanglement for β-VAE for β = 60 (though at the cost of much worse reconstruction where every generated sample looks like a hazy blob). On the other hand, we find the proposed SAP score to be correlated well with the qualitative disentanglement seen in the latent traversal plots. This is reflected in the higher SAP score of β-VAE for β = 60 than DIP-VAE-I. We also observe that for 2D Shapes data, DIP-VAE-II gives a much better trade-off between disentanglement (measured by the SAP score) and reconstruction error than both DIP-VAE-I and β-VAE, as shown quantitatively in Fig. 1 and qualitatively in the latent traversal plots in FIG0. The reason is that DIP-VAE-I enforces [Cov p(x) [µ φ (x)]] ii to be close to 1 and this may affect the disentanglement adversely by splitting a generative factor across multiple latents for 2D Shapes where the generative factors are much less than the latent dimension. For real datasets having lots of factors with complex generative processes, such as CelebA, DIP-VAE-I is expected to work well which can be seen in Fig. 2 where DIP-AVE-I yields a much lower reconstruction error with a higher SAP score (as well as higher Z-diff scores).Binary attribute classification for CelebA. We also experiment with predicting the binary attribute values for each test example in CelebA from the inferred µ φ (x). For each attribute k, we compute the attribute vector set, and project the µ φ (x) along these vectors. A bias is learned on these scalars (by minimizing hinge loss) which is then used for classifying the test examples. TAB1 shows the for the attribute which show the highest change across various methods (most other attribute accuracies do not change). The proposed DIP-VAE outperforms both VAE and β-VAE for most attributes. The performance of β-VAE gets worse as β is increased further. DISPLAYFORM0 Adversarial autoencoder BID40 also matches q φ (z) (which is referred as aggregated posterior in their work) to the prior p(z). However, adversarial autoencoder does not have the goal of minimizing KL(q φ (z|x)||p θ (z|x)) which is the primary goal of variational inference. It DISPLAYFORM0, where D is the distance induced by a discriminator that tries to classify z ∼ q φ (z) from z ∼ p(z) by optimizing a cross-entropy loss (which induces JS-divergence as D). This can be contrasted with the objective in.Invariance and Equivariance. Disentanglement is closely connected to invariance and equivariance of representations. If R: x → z is a function that maps the observations to the feature representions, equivariance (with respect to T) implies that a primitive transformation T of the input in a corresponding transformation T of the feature, i.e., R(T (x)) = T (R(x)). Disentanglement requires that T acts only on a small subset of dimensions of R(x) (a sparse action). In this sense, equivariance is a more general notion encompassing disentanglement as a special case, however this special case carries additional benefits of interpretability, ease of transferrability, etc. Invariance is also a special case of equivariance which requires T to be identity for R to be invariant to the action of T on the input observations. However, invariance can obtained more easily from disentangled representations than from equivariant representations by simply marginalizing the appropriate subset of dimensions. There exists a lot of prior work in the literature on equivariant and invariant feature learning, mostly under the supervised setting which assumes the knowledge about the nature of input transformations (e.g., rotations, translations, scaling for images, etc.) BID52 BID8 BID0 BID50 BID12 BID16 BID27 BID44 BID47. We proposed a principled variational framework to infer disentangled latents from unlabeled observations. Unlike β-VAE, our variational objective does not have any conflict between the data log-likelihood and the disentanglement of the inferred latents, which is reflected in the empirical . We also proposed the SAP disentanglement metric that is much better correlated with the qualitative disentanglement seen in the latent traversals than the Z-diff score. An interesting direction for future work is to take into account the sampling biases in the generative process, both natural (e.g., sampling the female gender makes it unlikely to sample beard for face images in CelebA) as well as artificial (e.g., a collection of face images that contain much more smiling faces for males than females misleading us to believe p(gender,smile) = p(gender)p(smile)), which makes the problem challenging and also somewhat less well defined (at least in the case of natural biases). Effective use of disentangled representations for transfer learning is another interesting direction for future work.

represent the tldr for finding the corresponding scientific article

We propose a variational inference based approach for encouraging the inference of disentangled latents. We also propose a new metric for quantifying disentanglement.

We show that posterior collapse in linear VAEs is caused entirely by marginal log-likelihood (not ELBO). Experiments on deep VAEs suggest a similar phenomenon is at play.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: In multiagent systems (MASs), each agent makes individual decisions but all of them contribute globally to the system evolution. Learning in MASs is difficult since each agent's selection of actions must take place in the presence of other co-learning agents. Moreover, the environmental stochasticity and uncertainties increase exponentially with the increase in the number of agents. Previous works borrow various multiagent coordination mechanisms into deep learning architecture to facilitate multiagent coordination. However, none of them explicitly consider action semantics between agents that different actions have different influences on other agents. In this paper, we propose a novel network architecture, named Action Semantics Network (ASN), that explicitly represents such action semantics between agents. ASN characterizes different actions' influence on other agents using neural networks based on the action semantics between them. ASN can be easily combined with existing deep reinforcement learning (DRL) algorithms to boost their performance. Experimental on StarCraft II micromanagement and Neural MMO show ASN significantly improves the performance of state-of-the-art DRL approaches compared with several network architectures. Deep reinforcement learning (DRL) has achieved a lot of success at finding optimal policies to address single-agent complex tasks . However, there also exist a lot of challenges in multiagent systems (MASs) since agents' behaviors are influenced by each other and the environment exhibits more stochasticity and uncertainties (; ; ;). Recently, a number of deep multiagent reinforcement learning (MARL) approaches have been proposed to address complex multiagent problems, e.g., coordination of robot swarm systems and autonomous cars . One major class of works incorporates various multiagent coordination mechanisms into deep multiagent learning architecture (; ;). proposed a centralized actor-critic architecture to address the partial observability in MASs. They also incorporate the idea of joint action learner (JAL) to facilitate multiagent coordination. proposed Counterfactual Multi-Agent Policy Gradients (COMA) motivated from the difference reward mechanism to address the challenges of multiagent credit assignment. proposed applying mean-field theory to solve large-scale multiagent learning problems. More recently, extended the idea of leniency to deep MARL and proposed the retroactive temperature decay schedule to address stochastic rewards problems. However, all these works ignore the natural property of the action influence between agents, which we aim to exploit to facilitate multiagent coordination. Another class of works focus on specific network structure design to address multiagent learning problems (; ; ;). designed a value-decomposition network (VDN) to learn an optimal linear value decomposition from the team reward signal based on the assumption that the joint actionvalue function for the system can be additively decomposed into value functions across agents. relaxed the linear assumption in VDN by assuming that the Q-values of individual agents and the global one are also monotonic, and proposed QMIX employing a network that estimates joint action-values as a complex non-linear combination of per-agent values. Recently, proposed the relational deep RL to learn environmental entities relations. However, they considered the entity relations on the pixel-level of raw visual data, which ignores the natural property of the influence of actions between agents. proposed a novel network architecture called Relational Forward Model (RFM) for predictive modeling in multiagent learning. RFM takes a semantic description of the state of an environment as input, and outputs either an action prediction for each agent or a prediction of the cumulative reward of an episode. However, RFM does not consider from the perspective of the influence of each action on other agents. OpenAI designed network structures to address multiagent learning problems in a famous Multiplayer Online Battle Arena (MOBA), Dota2. They used a scaled-up version of PPO , adopted the attention mechanism to compute the weight of choosing the target unit, with some of information selected from all information as input. However, this selection is not considered from the influence of each action on other agents. There are also a number of works designing network structures for multiagent communication . However, none of the above works explicitly leverage the fact that an agent's different actions may have different impacts on other agents, which is a natural property in MASs and should be considered in the decision-making process. In multiagent settings, each agent's action set can be naturally divided into two types: one type containing actions that affect environmental information or its private properties and the other type containing actions that directly influence other agents (i.e., their private properties). Intuitively, the estimation of performing actions with different types should be evaluated separately by explicitly considering different information. We refer to the property that different actions may have different impacts on other agents as action semantics. We can leverage the action semantics information to improve an agent's policy/Q network design toward more efficient multiagent learning. To this end, we propose a novel network architecture, named Action Semantics Network (ASN) to characterize such action semantics for more efficient multiagent coordination. The main contributions of this paper can be summarized as follows: 1) to the best of our knowledge, we are the first to explicitly consider action semantics and design a novel network to extract it to facilitate learning in MASs; 2) ASN can be easily combined with existing DRL algorithms to boost its learning performance; 3) experimental * on StarCraft II micromanagement and Neural MMO show our ASN leads to better performance compared with state-of-the-art approaches in terms of both convergence speed and final performance. Stochastic games (SGs) are a natural multiagent extension of Markov decision processes (MDPs), which models the dynamic interactions among multiple agents. Considering the fact that agents may not have access to the complete environmental information, we follow previous work's settings and model the multiagent learning problems as partially observable stochastic games (POSGs) . A partially observable stochastic game (POSG) is defined as a tuple N, S, where N is the set of agents; S is the set of states; A i is the set of actions available to agent i (the joint action space T is the transition function that defines transition probabilities between global states: S × A × S →; R i is the reward function for agent i: S × A → R and O i is the set of observations for agent i. Note that a state s ∈ S describes the environmental information and the possible configurations of all agents, while each agent i draws a private observation o i correlated with the state: S → O i, e.g., an agent's observation includes the agent's private information and the relative distance between itself and other agents. Formally, an observation of agent i at step t can be constructed as follows: is the observed environmental information, m i t is the private property of agent i (e.g., in robotics, m i t includes agent i's location, the battery power and the healthy status of each component) and the rest are the observations of agent i on other agents (e.g., in robotics, o i,i−1 t includes the relative location, the exterior of agent i−1 that agent i observes). An policy π i: O i ×A i → [0; 1] specifies the probability distribution over the action space of agent i. The goal of agent i is to learn a policy π i that maximizes the expected return with a discount factor γ: 3 THE ACTION SEMANTICS NETWORK ARCHITECTURE In MASs, multiple agents interact with the environment simultaneously which increases the environmental stochasticity and uncertainties, making it difficult to learn a consistent globally optimal policy for each agent. A number of Deep Multiagent Reinforcement Learning (MARL) approaches have been proposed to address such complex problems in MASs by either incorporating various multiagent coordination mechanisms into deep multiagent learning architecture; ) or designing specialized network structures to facilitate multiagent learning (; ;). However, none of them explicitly consider extracting action semantics, which we believe is a critical factor that we can leverage to facilitate coordination in multiagent settings. Specifically, each agent's action set can be naturally classified into two types: one type containing actions that directly affect environmental information or its private properties and the other type of actions directly influence other agents. Therefore, if an agent's action directly influences one of the other agents, the value of performing this action should be explicitly dependent more on the agent's observation for the environment and the information of the agent to be influenced by this action, while any additional information (e.g., part of the agent's observation for other agents) is irrelevant and may add noise. We refer to the property that different actions may have different impacts on other agents as action semantics. However, previous works usually use all available information for estimating the value of all actions, which can be quite inefficient. To this end, we propose a new network architecture called Action Semantics Network (ASN) that explicitly considers action semantics between agents to improve the estimation accuracy over different actions. Instead of inputting an agent's total observation into one network, ASN consists of several sub-modules that take different parts of the agent's observation as input according to the semantics of actions. In this way, ASN can effectively avoid the negative influence of the irrelevant information, and thus provide a more accurate estimation of performing each action. Besides, ASN is general and can be incorporated into existing deep MARL frameworks to improve the performance of existing DRL algorithms. In the next section, we will describe the ASN structure in detail. Considering the semantic difference of different actions, we classify an agent's action set A i of agent i into two subsets: A i in and A i out. A i in contains actions that affect the environmental information or its private properties and do not influence other agents directly, e.g., moving to different destinations would only affect its own location information. A i out corresponds to those actions that directly influence some of other agents, e.g., attack agent j in competitive settings or communicate with agent j in cooperative settings. Following the above classification, the proposed network architecture, ASN, explicitly considers the different influence of an agent's actions on other agents by dividing the network into different sub-modules, each of which takes different parts of the agent's observation as input according to the semantics of actions (shown in Figure 1). Considering an agent i and n − 1 agents in its neighborhood, ASN decouples agent i's network into n sub-modules as follows. The first one shown in Figure 1 (left side O2A i) contains a network O2E i which is used to generate the observation embedding e i given the full observation o i t of agent i as input, and a network E2A i (embedding to action) which generates the values of all action in A i in as output. The rest of n − 1 sub-modules (O2A i,j, j ∈ N, j = i) are used to estimate the values of those actions in A i out related with each influenced agent, composed of n − 1 networks (O2E i,j, j ∈ N, j = i) which are responsible for determining the observation embeddings related with each influenced agent, denoted as e i,j. Each of n − 1 sub-modules O2A i,j only takes a part of agent i's observation related with one neighbor agent j, o i,j t as input. 5 6 78 9 or = 78 9, 6, 6 ∈? 9@ 9 5 6 78 9 or = 78 9, 6, 6 ∈? 7A8 9! 2# $ Figure 1: ASN of agent i contains n sub-modules: i,n, each of which takes different parts of the agent's observation as input. For value-based RL methods, at each step t, the evaluation of executing each action a Next, we describe how ASN can be incorporated into existing deep MARL, which can be classified into two paradigms: Independent Learner (IL) and Joint Action Learner (JAL) (; ; . IL applies a single-agent learning algorithm to a multiagent domain to treat other agents as part of the environment. In contrast, JALs observe the actions of other agents, and optimize the policy for each joint action. Following the above two paradigms, we propose two classes of ASN-based MARL: ASN-IL and ASN-JAL. For ASN-IL, we focus on the case of combing ASN with PPO , a popular single-agent policy-based RL. The way ASN combines with other single-agent RL is similar. In contrast, ASN-JAL describes existing deep MARL approaches combined with ASN, e.g., QMIX and VDN . ASN-PPO For each agent i equipped with a policy network parameterized by θ i, ASN-PPO replaces the vanilla policy network architecture with ASN and optimizes the policy following PPO. Generally, IL ignores the existence of other agents, thus, for each agent i, the expected return J(θ i) is optimized using the policy gradient theorem: where A t is the advantage function at timestep t. PPO uses constraints and advantage estimation to reformulate the optimization problem as: where r t (θ i) is the probability ratio, θ i old is the policy parameters before the update. Then in ASN-PPO, r t (θ i) can be rewritten as follows by substituting Equation 2: Lastly, ASN-PPO maximizes the objective (Equation 3) following PPO during each iteration. ASN-QMIX The way ASN combines with deep MARL algorithms is similar and we use QMIX as an example to present. Figure 2 illustrates the ASN-QMIX network structure, where for each agent i, ASN-QMIX replaces the vanilla Q-network architecture with ASN. At each step t, the individu- is first calculated following Section 3.2 and then input into the mixing network. The mixing network mixes the output of all agents' networks monotonically and produces the joint action-value function Q tot (s t, a t). The weights of the mixing network are restricted to be non-negative and produced by separate hypernetworks, each of which takes state s t as input and generates the weights of one layer of the mixing network. Finally, ASN-QMIX is trained to minimize the loss: 2, where B is the batch size of transitions, y tot t = r t + γ max a Q tot (s, a ; θ −), and θ − are the parameters of the target network as in DQN . Multi-action ASN The general case in MASs is that an agent may have multiple actions which can directly influence another agent, e.g., a router can send packages with different size to one of its neighbors, a soldier can select different weapons to attack enemies and cause different damages. To address this, we extend the basic ASN to a generalized version, named Multi-action ASN (shown in Figure 3(a) ), that takes o i,j as input, and produces a number of embeddings e i,j1, · · ·, e i,jm, where m is the number of actions that directly influences agent j. After that, multi-action ASN calculates the estimation of performing each action, which uses a pairwise interaction function M to combine the two embeddings e i,j k,k∈ [1,m] and e i following Equation. Parameter-sharing between sub-modules Parameter-sharing (PS) mechanism is widely used in MARL. If agents are homogeneous, their policy networks can be trained more efficiently using PS which greatly reduces the training complexity . Recent work also incorporates PS on heterogeneous agents by adding extra information to identify agent type. Following previous work, here we incorporate PS to enable parameter-sharing between different sub-modules of ASN. The basic ASN (Figure 1) for agent i contains a number of sub-modules O2A i,j, each of which takes o i,j as input. In this way, if an action a i,j t ∈ A i out has a direct impact on any of another agent j, the number of sub-modules is equal to the number of other agents. The training of basic ASN is inefficient since the number of sub-modules is increasing with the increase in the number of agents. If the other agents that agent i can directly influence are homogeneous, the sub-module parameters can be shared across those agents. Thus, in a homogeneous MAS, all influencing agents can share one sub-module (shown in Figure 3 (b) ); in a MAS that contains several types of agents, each type of agents can share one sub-module (Mixed ASN in Figure 3 (c) ). Note that the basic ASN can be seen as the simplest case that designs a sub-module for each influencing agent without PS. We evaluate the performance of ASN compared with different network structures including the vanilla network (i.e., aggregate all information and input into one single network), the dueling network , the attention network that expects to learn which information should be focused on more automatically (i.e., adds an additional hidden layer to compute the weights of the input and then generate an element-wise product to input into the next layer) and entity-attention network (i.e., instead of computing attention weight for each dimension of the input, the weight is computed for each entity/agent) under various DRL approaches. Other network architectures as we mentioned before are not comparable here since they are orthogonal to our ASN. Our test domains include StarCraft II micromanagement and Massively Multiplayer Online Role-Playing Games (Neural MMO) . The details of neural network structures and parameter settings are in the appendix. StarCraft II is a real-time strategic game with one or more humans competing against each other or a built-in game AI. Here we focus on a decentralized multiagent control that each of the learning agents controls an individual army entity. At each step, each agent observes the local game state which consists of the following information for all units in its field of view: relative distance between other units, the position and unit type (detailed in the appendix) and selects one of the following actions: move north, south, east or west, attack one of its enemies, stop and the null action. Agents belonging to the same side receive the same joint reward at each time step that equals to the total damage on the enemy units. Agents also receive a joint reward of 10 points after killing each enemy, and 200 points after killing all enemies. The game ends when all agents on one side die or the time exceeds a fixed period. Note that previous works; ) reduce the learning complexity by manually adding a rule that forbids each agent to select an invalid action, e.g., attack an opponent that beyond the attack range and move beyond the grid border. We relax this setting since it requires prior knowledge, which is hard to obtain in the real world. We are interested in evaluating whether these rules can be learned automatically through end-to-end training as well. Thus, the following are based on the setting that each agent can select an action that causes an invalid effect, and in , the agent will standstill at the current time step. We also evaluate ASN following previous settings (adding the manual rule in StarCraft II that forbidding the invalid actions) and ASN still achieves better performance which can be found in the appendix. In StarCraft II 8m map (8 Marines vs 8 Marines), each agent is homogeneous to each other, so we adopt homogeneous ASN to evaluate whether it can efficiently characterize action semantics between two agents. Figure 4 input of different sub-modules, ASN enables an agent to learn the right timing to attack different opponents to maximize its total damage on opponents. In contrast, existing network architectures simply input all information into one network, thus an agent cannot distinguish the difference of effects that different actions may have on the opponents and may choose the suboptimal opponent to attack, thus ing in lower performance than ASN. Attention network performs better than vanilla and dueling when combined with IQL, while both of them show very similar performance with the vanilla network when combined with QMIX and VDN. However, entity-attention performs worst since it is hard to figure out the useful information for each entity when input all information into one network initially. Since the performance difference of other network architecture is marginal, we only present of ASN-QMIX compared with the vanilla network under QMIX (denoted as vanilla-QMIX) in the following sections. Figure 5 (a) we can observe that Mixed ASN-QMIX perform better than vanilla-QMIX. The reason is that ASN efficiently identifies action semantics between each type of two agents, thus it selects more proper attack options each time and achieves better performance last vanilla-QMIX. Is ASN still effective on large-scale scenarios? We further test on a large-scale agent space on a 15m map. Figure 5 (b) depicts the dynamics of the average win rates of ASN-QMIX and vanilla-QMIX. We can see that ASN-QMIX quickly learns the average win rates of approximately 80 %, while vanilla-QMIX fails, with the average win rates of approximately only 20 %. From Figure 4 (b) and 5 (b) we can find that with the increase of the agent number, the margin becomes larger between two methods. Intuitively, ASN enables an agent to explicitly consider more numbers of other agents' information with a larger agent size. However, for the vanilla network, it is more difficult to identify the action influence on other agents from a larger amount of mixed information, which in lower average win rates than ASN. An interesting observation for vanilla-QMIX is that they will learn to run away to avoid all being killed, and testing videos can be found in our anonymous website *. Can ASN recognize the influence of different actions? remove the manually added rule (which prevents selecting any invalid action), and agents would probably select the invalid action and standstill, which increases the learning difficulties. We can see that ASN-QMIX achieves an average percentage of approximately 71.9% for choosing a valid action. However, vanilla-QMIX only achieves an average percentage of approximately 44.3%. This phenomenon confirms that ASN effectively exploits action semantics between agents and enables agents to learn which action can be chosen at each time step, facilitating more robust learning, even in large-scale MASs. Can ASN effectively improve the estimation accuracy of actions? We investigate whether ASN can efficiently characterize the action semantics and facilitate multiagent coordination. To make the analysis more clear, we test the model learned on a 15m map on two illustrating scenarios: 1) the one-on-one combat scenario that the distance between two agents is dynamically changing; 2) the one Marine vs two Marines scenario that the HPs (Hit Points) of two opponents are dynamically different. Figure 6(a) shows the dynamics of the attack action's Q-value with the distance change of the ASN agent and its opponent. We can observe that the Q-value of the action that the ASN agent attacking its opponent decreases as the distance of the agent and its opponent increases, and stabilizes when the distance exceeds the attack range. However, the vanilla agent keeps the Qvalue of the attack action nearly unchanged. This indicates that ASN can automatically learn the information of when an action is valid and behave appropriately, while the vanilla agent has to rely on manually added rules to avoid choosing invalid actions. Figure 6 (b) and (c) shows the dynamics of the attack action's Q-value of ASN agent and vanilla agent with the HPs difference of two opponents changing (i.e., the HP difference equals to the HP of opponent 1 minus the HP of opponent 2). We can see that the ASN agent holds a higher Q-value of attacking opponent 1 when opponent 1's HP is lower than opponent 2 and vice versa. The symmetric curve of ASN is due to the fact that the state description of two opponents is very similar in this scenario. However, the vanilla agent always keeps a higher attack action's Q-value on Opponent 1 than on Opponent 2, which means it always selects to attack Opponent 1. These indicate that ASN can effectively exploit the action semantics between agents and improves the estimation accuracy on different actions, thus facilitates robust learning among agents. Does ASN exploits the 0-padding information? When one of the army units dies or some units are beyond the range of vision, one common practice is to use 0-paddings as the input for the observation of the died army unit. In this section, we provide an ablation study on whether ASN design 0.0 0.2 0.4 0.6 0.8 1.0 Step ×10 Step ×10 exploits the 0-padding information. Figure 7 shows the win rates of various network architectures combined with QMIX when using 1-paddings and −1-paddings as the input for the observation of the died army unit. We can see that ASN still performs best among all network architectures in terms of both convergence speed and final win rates. This indicates that ASN effectively extracts the action semantics between agents, instead of benefiting from the particular settings of 0-paddings. The Neural MMO is a massively multiagent environment that defines combat systems for a large number of agents. Figure 8 illustrates a simple Neural MMO scene with two groups of agents on a 10×10 tile. Each group contains 3 agents, each of which starts at any of the tiles, with HP = 100. At each step, each agent loses one unit of HP, observes local game state (detailed in the appendix) and decides on an action, i.e., moves one tile (up, right, left, down and stop) or makes an attack using any of three attack options (shown in the left part in Figure 8 : "Melee" with the attack distance is 2, the amount of damage is 5; "Range" with the attack distance is 4, the amount of damage is 2; "Mage" with the attack distance is 10, the amount of damage is 1). Each action that causes an invalid effect (e.g., attack an opponent that beyond the attack range and move beyond the grid border) would make the agent standstill. Each agent gets a penalty of −0.1 if the attack fails. The game ends when all agents in one group die, and agents belonging to the same group receive a joint reward, which is the difference of the total HPs between itself and its opposite side. In Neural MMO, an agent can attack one of its opponent using one of three different attack options, which can be used to evaluate whether multi-action ASN can efficiently identify the multiple action semantics between agents. Here we adopt two kinds of multi-action ASN: ASN-M1 that shares parameters of the first neural network layer across three attack actions on one enemy (as shown in Figure 3 and A2C ). We can observe that ASN performs best under all three IL approaches in terms of average rewards. This is because ASN can learn to choose appropriate actions against other agents at different time steps to maximize the damage on others. However, the vanilla network just mixes all information together which makes it difficult to identify and take advantage of the action semantics between agents, thus it achieves lower performance than ASN. Since the information is mixed initially, although the attention and entity-attention networks try to learn which information should be focused on more, it is hard to distinguish which part of the information is more useful, thus achieving lower performance than ASN. Can ASN recognize the best actions from multiple ones? We further investigate whether ASN can efficiently exploit different action semantics between agents and enable an agent to identify the best attack option (i.e., an attack that causes the most damage) with the distance between the agent and its opponent changing. Figure 10 shows the average attack damage of each attack option in Neural MMO when the distance between agent i and its opponent j is less than or equal to 2 (d ij ≤ 2). The best attack option is "Melee" within this distance range since it causes the maximum damage among three attacks. We can see that both ASN-M1 agent and ASN-M cause higher total damage than other methods, and ASN-M1 agent causes the highest total damage on average. However, the attention network only causes average total damage of approximately 1.5, the entity-attention and vanilla network only cause average total damage of approximately 1.0 due to the lower probability of selecting the best attack action "Melee". This is because two kinds of ASN have a larger probability to select the best attach option "Melee" than other two networks, thus causing larger total damage. Similar on other distance ranges (d i,j ≤ 4, d i,j ≤ 10) can be found in the appendix that ASN always causes higher total damage than other networks. We propose a new network architecture, ASN, to facilitate more efficient multiagent learning by explicitly investigating the semantics of actions between agents. To the best of our knowledge, ASN is the first to explicitly characterize the action semantics in MASs, which can be easily combined with various multiagent DRL algorithms to boost the learning performance. ASN greatly improves the performance of state-of-the-art DRL methods compared with a number of network architectures. In this paper, we only consider the direct action influence between any of two agents. As future work, it is worth investigating how to model the action semantics among more than two agents. Another interesting direction is to consider the action semantics between agents in continuous action spaces. 0.2 0.4 0.6 0.8 1.0 1.2 1.4 Step Here we provide the hyperparameters for StarCraft II † shown in Table 2. The following present the performance of ASN-QMIX and vanilla-QMIX under different StarCraft II maps with adding the manual rule (forbids the agent to choose the invalid actions). In a 10x10 tile (where each tile can be set as different kinds, e.g., rocks, grass), there are two teams of agents (green and red), each of which has 3 agents. At the beginning of each episode, each agent appears on any of the 10x10 tiles. The observation of an agent is in the form of a 43-dimensional vector, in which the first 8 dimensions are: time to live, HP, remaining foods (set 0), remaining water (set 0), current position (x and y), the amount of damage suffered, frozen state (1 or 0); the rest of 35 dimensions are divided equally to describe the other 5 agents' information. The first 14 dimensions describe the information of 2 teammates, following with the description of 3 opponents' information. Each observed agent's information includes the relative position(x and y), whether it is a teammate(1 or 0), HP, remaining foods, remaining water, and the frozen state. Each agent chooses an action from a set of 14 discrete actions: stop, move left, right, up or down, and three different attacks against one of its opponent ("Melee" with the attack distance is 2, the amount of damage is 5; "Range" with the attack distance is 4, the amount of damage is 2; "Mage" with the attack distance is 10, the amount of damage is 1). Each agent gets a penalty of −0.1 if the attack fails. They get a −0.01 reward for each tick and a −10 penalty for being killed. The game ends when a group of agents dies or the time exceeds a fixed period, and agents belonging to the same group receive the same reward, which is the difference of the total number of HPs between itself and its opposite side. The details of vanilla, attention, entity-attention networks for Neural MMO are shown in Figure 13 (a-c) which contains an actor network, and a critic network. All actors are similar to those for StarCraft II in Figure 11, except that the GRU layer is excluded and the output is the logic probability of choosing each action. All critics are the same as shown in Figure 13 (a). Since in Neural MMO, each agent has multiple actions that have direct influence on each other agent, i.e., three kinds of attack options, we test two kinds of ASN variants: one (Figure 13(d) ) is the Multi-action ASN we mentioned in the previous section that shares the first layer parameters among multiple actions; the other (Figure 13 (e)) is the basic homogeneous ASN that does not share the first layer parameters among multiple actions. Here we provide the hyperparameters for Neural MMO shown in Table 3. The above present the average attack damage of each attack option under the different distance ranges between the agent and its opponent.

Represent the TLDR for finding the corresponding scientific article

Our proposed ASN characterizes different actions' influence on other agents using neural networks based on the action semantics between them.

Represent the short summary for finding the corresponding scientific article (long):

Techniques for combining generalized policies with search algorithms to exploit the strengths and overcome the weaknesses of each when solving probabilistic planning problems


Question: Represent the scientific article to find a short TLDR

Question: Spiking neural networks are being investigated both as biologically plausible models of neural computation and also as a potentially more efficient type of neural network. While convolutional spiking neural networks have been demonstrated to achieve near state-of-the-art performance, only one solution has been proposed to convert gated recurrent neural networks, so far. Recurrent neural networks in the form of networks of gating memory cells have been central in state-of-the-art solutions in problem domains that involve sequence recognition or generation. Here, we design an analog gated LSTM cell where its neurons can be substituted for efficient stochastic spiking neurons. These adaptive spiking neurons implement an adaptive form of sigma-delta coding to convert internally computed analog activation values to spike-trains. For such neurons, we approximate the effective activation function, which resembles a sigmoid. We show how analog neurons with such activation functions can be used to create an analog LSTM cell; networks of these cells can then be trained with standard backpropagation. We train these LSTM networks on a noisy and noiseless version of the original sequence prediction task from , and also on a noisy and noiseless version of a classical working memory reinforcement learning task, the T-Maze. Substituting the analog neurons for corresponding adaptive spiking neurons, we then show that almost all ing spiking neural network equivalents correctly compute the original tasks. With the manifold success of biologically inspired deep neural networks, networks of spiking neurons are being investigated as potential models for computational and energy efficiency. Spiking neural networks mimic the pulse-based communication in biological neurons, where in brains, neurons spike only sparingly -on average 1-5 spikes per second BID0. A number of successful convolutional neural networks based on spiking neurons have been reported BID7 BID13 BID6 BID15 BID12, with varying degrees of biological plausibility and efficiency. Still, while spiking neural networks have thus been applied successfully to solve image-recognition tasks, many deep learning algorithms use recurrent neural networks (RNNs), in particular using Long Short-Term Memory (LSTM) layers BID11. Compared to convolutional neural networks, LSTMs use memory cells to store selected information and various gates to direct the flow of information in and out of the memory cells. To date, the only spike-based version of LSTM has been realized for the IBM TrueNorth platform Shrestha et al.: this work proposes a method to approximate LSTM specifically for the constrains of this neurosynaptic platform by means of a store-and-release mechanism that synchronizes the modules. This translates to a frame-based rate coding computation, which is less biological plausible and energy efficient than an asynchronous approach, as the one proposed here. Here, we demonstrate a gated recurrent spiking neural network that corresponds to an LSTM unit with a memory cell and an input gate. Analogous to recent work on spiking neural networks (O '; BID6 BID19 BID20, we first train a network with modified LSTM units that computes with analog values, and show how this LSTMnetwork can be converted to a spiking neural network using adaptive stochastic spiking neurons that encode and decode information in spikes using a form of sigma-delta coding BID18 BID19 BID14 . In particular, we develop a binary version of the adaptive sigma-delta coding proposed in BID19 : we approximate the shape of the transfer function that this model of fast-adapting spiking neurons exhibits, and we assemble the analog LSTM units using just this transfer function. Since input-gating is essential for maintaining memorized information without interference from unrelated sensory inputs BID11, and to reduce complexity, we model a limited LSTM neuron consisting of an input cell, input gating cell, a Constant Error Carousel (CEC) and output cell. The ant analog LSTM network is then trained on a number of classical sequential tasks, such as the noise-free and noisy Sequence Prediction and the T-Maze task BID11 BID1. We demonstrate how nearly all the corresponding spiking LSTM neural networks correctly compute the same function as the analog version. Note that the conversion of gated RNNs to spike-based computation implies a conversion of the neural network from a time step based behavior to the continuous-time domain: for RNNs, this means having to consider the continuous signal integration in the memory cell. We solve the time conversion problem by approximating analytically the spiking memory cell behavior through time. Together, this work is a first step towards using spiking neural networks in such diverse and challenging tasks like speech recognition and working memory cognitive tasks. To construct an Adapting Spiking LSTM network, we first describe the Adaptive Spiking Neurons and we approximate the corresponding activation function. Subsequently, we show how an LSTM network comprised of a spiking memory cell and a spike-driven input-gate can be constructed and we discuss how analog versions of this LSTM network are trained and converted to spiking versions. Adaptive Spiking Neuron. The spiking neurons that are used in this paper are Adaptive Spiking Neurons (ASNs) as described in BID2. This is a variant of an adapting Leaky Integrate & Fire (LIF) neuron model that includes fast adaptation to the dynamic range of input signals. The ASNs used here communicate with spikes of a fixed height h = 1 (binary output), as suggested by BID20. The behavior of the ASN is determined by the following equations: incoming postsynaptic current: DISPLAYFORM0 input signal: DISPLAYFORM1 threshold: DISPLAYFORM2 internal state: DISPLAYFORM3 where w i is the weight (synaptic strength) of the neuron's incoming connection; t i s < t denote the spike times of neuron i, and t s < t denote the spike times of the neuron itself; φ(t) is an exponential smoothing filter with a short time constant τ φ; ϑ 0 is the resting threshold; m f is a variable controlling the speed of spike-rate adaptation; τ β, τ γ, τ η are the time constants that determine the rate of decay of I(t), ϑ(t) andŜ(t) respectively (see BID2 and BID19 for more details). As in BID2, the ASN emits spikes following a stochastic firing condition defined as: DISPLAYFORM4 where V (t) is the membrane potential defined as the difference between S(t) andŜ(t), λ 0 = 0.005 is a normalization parameter and ∆V = 0.1 is a scaling factor that defines the slope of the stochastic area. Activation function of the Adaptive Analog Neuron. In order to create a network of ASNs that performs correctly on typical LSTM tasks, our approach is to train a network of Adaptive Analog Neurons (AANs) and then convert the ing analog network into a spiking one, similar to O'; BID6; BID19. We define the activation function of the AANs as the function that maps the input signal S to the average PSC I that is perceived by the next (receiving) ASN in a defined time window. We normalize the obtained spiking activation function at the point where it reaches a plateau. We then fit the normalized spiking activation function with a sum-of-exponentials shaped function as: DISPLAYFORM5 with derivative: DISPLAYFORM6 where, for the neuron parameters used, we find a = 148.7, b = −10.16, c = 3.256 and d = −1.08.Using this mapping from the AAN to the ASN (see Figure 1), the activation function can be used during training: thereafter, the ASNs are used as "drop in" replacements for the AANs in the trained network. Unless otherwise stated, the ASNs use τ η = τ β = τ γ = 10 ms, and ϑ 0 and m f are set to 0.3 and 0.18 for all neurons. The spike height, h, is found such that ASN(4.8) = 1. Note that the spike height h is a normalization parameter for the activation function of the ASN model: in order to have binary communication across the network, the output weights are simply scaled by h. Adaptive Spiking LSTM. An LSTM cell usually consists of an input and output gate, an input and output cell and a CEC BID11. Deviating from the original formulation, and more recent versions where forget gates and peepholes were added Gers et al. FORMULA1, the Adaptive Spiking LSTM as we present it here only consists of an input gate, input and output cells, and a CEC. As noted, to obtain a working Adaptive Spiking LSTM, we first train its analog equivalent, the Adaptive Analog LSTM. Figure 2 shows the schematic of the Adaptive Analog LSTM and its spiking analogue. It is important to note that we aim for a one-on-one mapping from the Adaptive Analog LSTM to the Adaptive Spiking LSTM. This means that while we train the Adaptive Analog LSTM network with the standard time step representation, the conversion to the continuous-time spiking domain is achieved by presenting each input for a time window of size ∆t. Sigmoidal ASN. The original formulation of LSTM uses sigmoidal activation functions in the input gate and input cell. However, the typical activation function of real neurons resembles a half-sigmoid and we find that the absence of a gradient for negative input is problematic during training. Here, we approximate a sigmoidal-shaped activation function by exploiting the stochastic firing condition of the ASN. Indeed, Figure 1 shows that the ASN has a non-null probability to fire even under the threshold ϑ 0. Therefore, the AAN transfer function of Eq. 6 holds a gradient in that area. Together with the maximal activation being normalized to 1 (see Eq. 6 for lim S→∞) the AAN transfer function represents a good candidate for LSTM operations such as closing and opening the gates. Spiking input gate and spiking input cell. The AAN functions are used in the Adaptive Analog LSTM cell for the input gate and input cell. The activation value of the input cell is multiplied by the activation value of the input gate, before it enters the CEC, see Figure 2. In the spiking version of the input gate, the outgoing signal from the ASN is accumulated in an intermediate neuron (ASN * in Figure 2). The internal stateŜ of this neuron is then multiplied with the spikes that move from the ASN of the input cell to the ASN of the output cell. This leads to a direct mapping from the Adaptive Analog LSTM to the Adaptive Spiking LSTM.Spiking Constant Error Carousel (CEC) and spiking output cell. The Constant Error Carousel (CEC) is the central part of the LSTM cell and avoids the vanishing gradient problem BID11. In the Adaptive Spiking LSTM, we merge the CEC and the output cell to one ASN with an internal state that does not decay -in the brain could be implemented by slowly decaying (seconds) neurons BID5. The value of the CEC in the Adaptive Analog LSTM corresponds with state I of the ASN output cell in the Adaptive Spiking LSTM.In the Adaptive Spiking LSTM, we set τ β in Equation 1 to a very large value for the CEC cell to obtain the integrating behavior of a CEC. Since no forget gate is implemented this in a spiking CEC neuron that fully integrates its input. When τ β is set to ∞, every incoming spike is added to a non-decaying PSC I. So if the state of the sending neuron (ASN in in FIG1) has a stable inter-spike interval (ISI), then I of the receiving neuron (ASN out) is increased with incoming spike height h every ISI, so h ISI per time step. For a stochastic neuron, this corresponds to the average increase per time step. The same integrating behavior needs to be translated to the analog CEC. Since the CEC cell of the Adaptive Spiking LSTM integrates its input S every time step by S τη, we can map this to the CEC of the Adaptive Analog LSTM. The CEC of a traditional LSTM without a forget gate is updated every time step by CEC(t) = CEC(t − 1) + S, with S its input value. The CEC of the Adaptive Analog LSTM is updated every time step by CEC(t) = CEC(t − 1) + S τη. This is depicted in Figure 2 via a weight after the input gate with value 1 τη. To allow a correct continuous-time representation after the spike-coding conversion, we divide the incoming connection weight to the CEC, W CEC, by the time window ∆t. In our approach then, we train the Adaptive Analog LSTM as for the traditional LSTM (without the τ η factor), which effectively corresponds to set a continuous-time time window ∆t = τ η. Thus, to select a different ∆t, in the spiking version W CEC has to be set to W CEC = τ η /∆t. The middle plot in FIG1 shows that setting τ β to ∞ for ASN out in a spiking network in the same behavior as using an analog CEC that integrates with CEC(t) = CEC(t − 1) + S, since the slope of the analog CEC is indeed the same as the slope of the spiking CEC. Here, every time step in the analog experiment corresponds to ∆t = 200 ms. However, the spiking CEC still produces an error with respect to the analog CEC (the error increases for lower ∆ts, e.g. it doubles when going from 200ms to 50ms). This is because of two reasons: first, the stochastic firing condition in an irregular ISI; second, the adapting behavior of the ASN produces a transitory response that is not represented by the AAN transfer function. For these reasons, by choosing bigger time windows ∆t more stable responses are obtained. Learning rule used for training the spiking LSTM To train the analog LSTMs on the supervised tasks, a customized truncated version of real-time recurrent learning (RTRL) was used. This is the same algorithm used in Gers et al. FORMULA1, where the partial derivatives w.r.t. the weights W xc and W xi (see Figure 2) are truncated. For the reinforcement learning (RL) tasks we used RL-LSTM Bakker FORMULA1, which uses the same customized, truncated version of RTRL that was used for the supervised tasks. RL-LSTM also incorporates eligibility traces to improve training and Advantage Learning BID10. All regular neurons in the network are trained with traditional backpropagation. Since the presented Adaptive Analog LSTM only has an input gate and no output or forget gate, we present four classical tasks from the LSTM literature that do not rely on these additional gates. Sequence Prediction with Long Time Lags. The main concept of LSTM, the ability of a CEC to maintain information over long stretches of time, was demonstrated in in a Sequence Prediction task: the network has to predict the next input of a sequence of p + 1 possible input symbols denoted as a 1,..., a p−1, a p = x, a p+1 = y. In the noise free version of this task, every symbol is represented by the p + 1 input units with the i − th unit set to 1 and all the others to 0. At every time step a new input of the sequence is presented. As in the original formulation, we train the network with two possible sequences, (x, a 1, a 2, ..., a p−1, x) and (y, a 1, a 2, ..., a p−1, y), chosen with equal probability. For both sequences the network has to store a representation of the first element in the memory cell for the entire length of the sequence (p). We train 20 networks on this task for a total of 100k trials, with p = 100, on an architecture with p + 1 input units and p + 1 output units. The input units are fully connected to the output units without a hidden layer. The same sequential network construction method from the original paper was used to prevent the "abuse problem": the Adaptive Analog LSTM cell is only included in the network after the error stops decreasing BID11. In the noisy version of the sequence prediction task, the network still has to predict the next input of the sequence, but the symbols from a 1 to a p−1 are presented in random order and the same symbol can occur multiple times. Therefore, only the final symbols a p and a p+1 can be correctly predicted. This version of the sequence prediction task avoids the possibility that the network learns local regularities in the input stream. We train 20 networks with the same architecture and parameters of the previous task, but for 200k trials. For both noise-free and noisy tasks we considered the network converged when the average error over the last 100 trials was less than 0.25.T-Maze task. In order to demonstrate the generality of our approach, we trained a network with Adaptive Analog LSTM cells on a Reinforcement Learning task, originally introduced in BID1. In the T-Maze task, an agent has to move inside a maze to reach a target position in order to BID11 and current implementation); while for the T-Maze tasks it corresponds to the total number of steps BID1. ASN accuracy (%), total number of spikes per task and firing rate (Hz) are also reported. Note that the firing rate for both the sequence prediction tasks are computed without taking into account the input and output neurons not active in a specific time frame. Task be rewarded while maintaining information during the trial. The maze is composed of a long corridor with a T-junction at the end, where the agent has to make a choice based on information presented at the start of the task. The agent receives a reward of 4 if it reaches the target position and −0.4 if it moves against the wall. If it moves to the wrong direction at the T-junction it also receives a reward of −0.4 and the system is reset. The larger negative reward value, w.r.t. the one used in BID1, is chosen to encourage Q-values to differentiate more during the trial. The agent has 3 inputs and 4 outputs corresponding to the 4 possible directions it can move to. At the beginning of the task the input can be either 011 or 110 (which indicates on which side of the T-junction the reward is placed).Here, we chose the corridor length N = 20. A noiseless and a noisy version of the task were defined: in the noiseless version the corridor is represented as 101, and at the T-junction 010; in a noisy version the input in the corridor is represented as a0b where a and b are two uniformly distributed random variables in a range of. While the noiseless version can be learned by LSTM-like networks without input gating BID16, the noisy version requires the use of such gates. The network consists of a fully connected hidden layer with 12 AAN units and 3 Adaptive Analog LSTMs. To increase the influence of the LSTM cell in the network, we normalized the activation functions of the AAN output cell and ASN output cell at S = 1. The same training parameters are used as in Bakker FORMULA1; we train 20 networks for each task and all networks have the same architecture. As a convergence criteria we checked whenever the network reached on average a total reward greater than 3.5 in the last 100 trials. As shown in TAB0, all of the networks that were successfully trained for the noise-free and noisy Sequence Prediction tasks could be converted into spiking networks. FIG3 shows the last 6 inputs of a noise-free Sequence Prediction task before (left) and after (right) the conversion, demonstrating the correct predictions made in both cases. Indeed, for the 19 successful networks, after presenting either x or y as the first symbol of the sequence, the average error over the last 200ms was always below the chosen threshold of 0.25. As it can be seen in Figure 6, the analog and the spiking CEC follow a comparable trend during the task, reaching similar values at the end of the simulation. Note that, in the noisy task, all the successfully trained networks were still working after the conversion: in this case, due to the input noise, the CEC values are always well separated. Finally, we found that the number of trials needed to reach the convergence criterion were, on average, lower than the one reported in BID11.Similar were obtained for the T-Maze task: all the networks were successful after the conversion in both the noise-free and noisy conditions. FIG4 shows the Q-values of a noisy T-Maze task, demonstrating the correspondence between the analog and spiking representation even in presence of noisy inputs. However, we notice that the CEC of the spiking LSTMs reach different values compared to their analog counterparts. This is probably due to the increased network and task complexity. In general, we see that the spiking CEC value is close to the analog CEC value, while always exhibiting some deviations. Moreover, TAB0 reports the average firing rate computed per task, showing reasonably low values compatible with the one recorder from real neurons. Gating is a crucial ingredient in recurrent neural networks that are able to learn long-range dependencies BID11. Input gates in particular allow memory cells to maintain information over long stretches of time regardless of the presented -irrelevant -sensory input BID11. The ability to recognize and maintain information for later use is also that which makes gated RNNs like LSTM so successful in the great many sequence related problems, ranging from natural language processing to learning cognitive tasks BID1.To transfer deep neural networks to networks of spiking neurons, a highly effective method has been to map the transfer function of spiking neurons to analog counterparts and then, once the network has been trained, substitute the analog neurons with spiking neurons; BID6; BID19. Here, we showed how this approach can be extended to gated memory units, and we demonstrated this for an LSTM network comprised of an input gate and a CEC. Hence, we effectively obtained a low-firing rate asynchronous LSTM network. The most complex aspect of a gating mechanism turned out to be the requirement of a differentiable gating function, for which analog networks use sigmoidal units. We approximated the activation function for a stochastic Adaptive Spiking Neurons, which, as many real neurons, approximates a half-sigmoid (Fig. 1). We showed how the stochastic spiking neuron has an effective activation even below the resting threshold ϑ 0. This provides a gradient for training even in that area. The ant LSTM network was then shown to be suitable for learning sequence prediction tasks, both in a noise-free and noisy setting, and a standard working memory reinforcement learning task. The learned network could then successfully be mapped to its spiking neural network equivalent for at least 90% of the trained analog networks. Figure 6: The values of the analog CECs and spiking CECs for the noise-free Sequence Prediction (left, only one CEC cell was used) and noise-free T-maze (right, three CEC cells were used) tasks. The spiking CEC is the internal stateŜ of the output cell of the Adaptive Spiking LSTM.We also showed that some difficulties arise in the conversion of analog to spiking LSTM. Principally, the ASN activation function is derived for steady-state adapted spiking neurons, and this difference causes an error that may be large for fast changing signals. Analog-valued spikes as explored in BID19 could likely resolve this issue, at the expense of some loss of representational efficiency. Although the adaptive spiking LSTM implemented in this paper does not have output gates BID11, they can be included by following the same approach used for the input gates: a modulation of the synaptic strength. The reasons for our approach are multiple: first of all, most of the tasks do not really require output gates; moreover, modulating each output synapse independently is less intuitive and biologically plausible than for the input gates. A similar argument can be made for the forget gates, which were not included in the original LSTM formulation: here, the solution consists in modulating the decaying factor of the CEC.Finally, which gates are really needed in an LSTM network is still an open question, with answers depending on the kind of task to be solved BID9 BID21. For example, the AuGMEnT framework does not use gates to solve many working memory RL tasks BID16. In addition, it has been shown by BID4; BID9 that a combination of input and forget gates can outperform LSTM on a variety of tasks while reducing the LSTM complexity.

Represent the short summary for finding the corresponding scientific article (long):

We demonstrate a gated recurrent asynchronous spiking neural network that corresponds to an LSTM unit.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We present eligibility propagation an alternative to BPTT that is compatible with experimental data on synaptic plasticity and competes with BPTT on machine learning benchmarks.