Question: Represent the article for finding a single-sentence TLDR

Question: The need for large amounts of training image data with clearly defined features is a major obstacle to applying generative adversarial networks(GAN) on image generation where training data is limited but diverse, since insufficient latent feature representation in the already scarce data often leads to instability and mode collapse during GAN training. To overcome the hurdle of limited data when applying GAN to limited datasets, we propose in this paper the strategy of \textit{parallel recurrent data augmentation}, where the GAN model progressively enriches its training set with sample images constructed from GANs trained in parallel at consecutive training epochs. Experiments on a variety of small yet diverse datasets demonstrate that our method, with little model-specific considerations, produces images of better quality as compared to the images generated without such strategy. The source code and generated images of this paper will be made public after review. Generative Adversarial Networks(GAN) BID5 ) are powerful unsupervised learning models that have recently achieved great success in learning high-dimensional distributions in various types of problems and on different datasets. In the context of image generation, the basic framework of a GAN model consists of two parts: a generator G that generates images by translating random input z into an image, and a discriminator D which determines the authenticity of a generated image x as compared to the real data. These two components are alternatively optimized against each other during the training process, with the goal of minimizing the difference between the distribution of generated image data and target distribution of real image data. A notable challenge in GAN training, however, lies in the need for large amounts of clearly labeled data to capture the diversity features across various types of images into the model. Such requirement makes it difficult or even impossible to utilize GAN in applications where the amount of available training data is small but diverse. Moreover, recent deep learning models BID6 ) have demonstrated tendencies of misrepresentation in classification tasks when influenced by adversarial noise. Such vulnerability may also translate to unsatisfactory image generation as most generative models are implemented with deep networks. Thus, given these considerations, we propose in this paper the strategy of parallel recurrent sample augmentation agnostic to specific model details. Our contributions can be summarized as follows:• We proposed a general black-box method using recurrent image addition to diversify training data and enhance its quality over a large class of GANs without model specifications.• We also includes in our model a novel K-fold parallel framework, which better augments training data by stabilizing model output and preventing overfitting.• Experiments across various datasets and GAN objectives demonstrate the effectiveness of our method using authenticity measures such as Inception Score and Frechet Inception Distance. Building reliable deep generative adversarial models on limited amounts of training data has been a persistent challenge within the research community. Previous efforts to address the issue of labeled data scarcity generally fall into two groups: optimizing the structures of GANs to allow for better feature representation of data, and augmenting the training data through techniques. Along the first line of research, prior research optimized the GAN in BID5 by considering stronger mathematical objectives for more powerful latent space representation in general BID14,, BID7, BID13 ). In addition, recent research on GANs BID8, BID14 ) reparametrized the input noise using variational inference by assuming that the latent space could be modeled by a tractable prior, but noise reparametrization has severe mathematical limitation that prevents applicability to more general models. Furthermore, distributed multi-discriminator models BID10, ) also enhance the performances, with great potential room for further optimization. For the second line of research, data augmentation has already enjoyed considerable success in the problem of image classification BID12 ). Traditional data augmentation methods such as crop, mirror, rotation and distortion BID12, BID17 ) generally require domain-specific expert knowledge and manual operations, and only produce limited variation in augmented images. Recent developments centered on automatic augmentation of training data using controlled RNNs after transforming the problem into policy search in reinforcement learning , but the space complexity the search algorithm requires still does not apply to quick augmentation with limited data. In this section we describe the details of Parallel Recurrent Data Augmentation(PRDA), which consists of recurrent image data construction via noise addition on images and parallel generation with fold division. Recent research suggests that adversarial noise to deep learning models greatly perturbs the performances of deep neural networks. In CNNs for classification, for instance, the neural network may assign irrelevant labels to semantically unambiguous images after the addition of noise BID6 ). This phenomeon is due to the omission of possible latent features in new data generation caused by over-dependency on our limited training data. Since virtually all GANs are implemented with deep networks, a similar deficiency in representation of latent feature space may translate to lower qualities of the generated images. To counter this effect, we consider the strategy of recurrent data augmentation, which constructs varied images given the limited training set by repeatedly generating and modifying these samples on training set for subsequent training sample generation. Running the original generative model for a fixed number of times, we extract sampled images using standard procedures of sample image generation as described in BID15, BID7. Random noise is then added to these samples to produce new images, which are then used for subsequent training. This procedure is repeated for a fixed number of times or until convergence. FIG0 is a flow-chart of our procedure. Notice that the addition of high dimensional normal random noise allows the additional images to retain the original latent features to be learned by GAN. Compared with traditional methods such as rotation, cropping and mirroring BID17, BID12 ) which may lead to information loss, random noise addition doesn't reduce the information about the latent features in training set while making the model more robust, because the expectation of noise is invariant at 0. Additionally, noise addition is agnostic to the type of generative model, since the procedure is independent from the specific choice of neural network or objective functions. Additionally, we introduce a parallel data generation strategy inspired by K-fold cross validation in machine learning BID2 ). Dividing the training data into K folds at the beginning, we run in parallel K independent generators on K data groups, each consisting of K − 1 folds of the training set. When data is generated in each generator, the sample images produced by each generator at the given epochs are then added with random noise. These noised images, in turn, are fed back into the respective training data sets. To allow for maximal usage of each generated image, we insert the images such that the image generated by one generator goes to the augmented training set of all other K − 1 generators. This is to insure that the different generators in parallel have access to as many varied data pieces as possible in subsequent steps of training, so as to prevent overfitting and bolster the robustness of our model. Figure 2 demonstrates the mechanism of our algorithm. Notice that our K-fold division goes hand in hand with the recurrent data generation with no need for model specific considerations. As demonstrated by our experiments Section 4, training different GANs in parallel from different folds of data substantially boosts the quality of the training set and that of the generated images. For a comparative analysis, we have conducted experiments on previous GAN architectures over various datasets with/without data augmentation. The GANs we have tested on include DC-GAN BID15 ), BEGAN BID1 ), and WGAN-GP BID7 ). Additionally, to simulate limited data, we randomly select 5000 images from the datasets CIFAR-10, CelebA and Places to create their corresponding reduced datasets named reduced-CIFAR, reducedCelebA, and reduced-Places, and conduct our experiments on these limited datasets. All of our experiments are conducted with CPU Intel(R) Core(R) CPU 8700-K (3.7GHz) and GPU GTX 1080. In our experiments, we augment the training set with 8 noised images every 100 training epoches, and repeat the procedure 3-5 times. By comparison, the unaugmented GAN is run over the same initial training data, with the number of epochs the same as the product of 100 and augmentation times. FIG1,4,5 are some sample images that our method produces with the state-of-the-art GAN WGAN-GP as compared to the ones produced by GAN without data augmentation. We observe that GANs with parallel recurrent image augmentation produce semantically coherent and visually diverse images earlier than the unaugmented GANs, while able to avoid fluctuations seen in unaugmented GANs during training. To evaluate the quality of the images generated by our augmentation method as compared with those generated without augmentation, we use the Inception Score(IS) BID16 ) and Frechet BID9 ). IS measures the entropy of generated images, with higher scores indicating greater diversity. On the other hand, FID measures the distance between the generated data and real data with two respective means and variances. Thus, the larger the IS and the smaller the FID, the better the performances of the model. Table 2: IS/FID scores of GANs on Reduced-CIFAR with/without Augmentation, given K = 5 Table 1 lists the combinations of GAN and dataset we tested our strategy on, as well as the Inception Score and Frechet Inception Distance of the images that are generated with and without our method using the state-of-the-art GAN WGAN-GP. Table 2 lists IS and FID of different GAN models on the Reduced-CIFAR with/without data augmentation. Clearly, on a variety of GAN structures and Datasets, recurrent sample augmentation produces better images as measured quantitatively. In sum, our paper shows that parallel recurrent sample augmentation can significantly improve the quality of synthetic images for a large class of GAN models. Our strategy is not only simple to implement, but also agnostic to the specific type of GAN to be improved on. As a further step, we are investigating the relationship between our proposed approach and other established methods. One possible pathway, for instance, lies in reinforcement learning as described in BID3 that gives more control to image generation via reward designation. We also hope to apply our idea to other generative models such as the VAE BID11 ) and further optimize our strategy using recent theoretical advances.

represent the tldr for finding the corresponding scientific article.
------
For example, Inspired by Information Bottleneck theory,  we propose a new architecture of GAN for a disentangled representation learning should be similar to We present a novel architecture of GAN for a disentangled representation learning. The new model architecture is inspired by Information Bottleneck (IB) theory thereby named IB-GAN. IB-GAN objective is similar to that of InfoGAN but has a crucial difference; a capacity regularization for mutual information is adopted, thanks to which the generator of IB-GAN can harness a latent representation in disentangled and interpretable manner. To facilitate the optimization of IB-GAN in practice, a new variational upper-bound is derived. With experiments on CelebA, 3DChairs, and dSprites datasets, we demonstrate that the visual quality of samples generated by IB-GAN is often better than those by β-VAEs. Moreover, IB-GAN achieves much higher disentanglement metrics score than β-VAEs or InfoGAN on the dSprites dataset. Learning good representations for data is one of the essential topics in machine learning community. Although any strict definition for it may not exist, the consensus about the useful properties of good representations has been discussed throughout many studies BID9; BID10. A disentanglement, one of those useful properties of representation, is often described as a statistical independence or factorization; each independent factor is expected to be semantically well aligned with the human intuition on the data generative factor (e.g. a chair-type from azimuth on Chairs dataset BID6, or age from azimuth on CelebA dataset ). The learned representation distilling each important factors of data into a single independent direction is hard to be done but highly valuable for many other downstream tasks (; b; .Many models have been proposed for disentangled representation learning (; ; ; ; BID13 . Despite their impressive , they either require knowledge of ground-truth generative factors or weak-supervision (e.g. domain knowledge or partial labels). In contrast, among many unsupervised approaches BID14;;; BID15 ), yet the two most successful approaches for the independent factor learning are β-VAE BID20 and InfoGAN BID12. BID20 demonstrate that encouraging the KL-divergence term of Variational autoencoder (VAE) objective by multiplying a constant β > 1 induces a high-quality disentanglement of latent factors. As follow-up research, BID10 provide a theoretical justification of the disentangling effect of β-VAE in the context of Information Bottleneck theory BID25 BID24 BID12 propose another fully unsupervised approach based on Generative Adversarial Network (GAN) BID18. He achieves the goal by enforcing the generator to learn disentangled representations through increasing the mutual information (MI) between the generated samples and the latent representations. Although InfoGAN can learn to disentangle representations for relatively simple datasets (e.g. MNIST, 3D Chairs), it struggles to do so on more complicated datasets such as CelebA. Moreover, the disentangling performance of the learned representations from InfoGAN is known as not good as the performance of the β-VAE and its variant models BID20; BID11.Stimulated by the success of β-VAE models BID10 BID20; BID11 BID17 with the Information Bottleneck theory BID5 BID0 ) in disentangled representations learning task, we hypothesize that the weakness of InfoGAN in the representation learning may originate from that it can only maximize the mutual information but lacks any constraining mechanisms. In other words, InfoGAN misses the term upper-bounding the mutual information from the perspective of IB theory. We present a novel unsupervised model named IB-GAN (Information Bottleneck GAN) for learning disentangled representations based on IB theory. We propose a new architecture of GANs from IB theory so that the training objective involves an information capacity constraint that InfoGAN lacks but β-VAE has. We also derive a new variational approximation algorithm to optimize IB-GAN objective in practice. Thanks to the information regularizer, the generator can use the latent representations in a manner that is both more interpretable and disentangled than InfoGANThe contributions of this work are summarized as follows:1. IB-GAN is a new GAN-based model for fully unsupervised learning of disentangled representations. To the best of our knowledge, there is no other unsupervised GAN-based model for this sake except the InfoGAN's variants BID20 ).2. Our work is the first attempt to utilize the IB theory into the GAN-based deep generative model. IB-GAN can be seen as an extension to the InfoGAN, supplementing an information constraining regularizer that InfoGAN misses.3. IB-GAN surpasses state-of-the-art disentanglement scores of BID20 BID16 on dSprites dataset . The quality of generated samples by IB-GAN on 3D Chairs BID6 and CelebA is also much realistic compared to that of the existing β-VAE variants of the same task. We remind some s: IB principle in section 2.1 and the connection between β-VAE and IB theory BID10 in section 2.2. Lastly, InfoGAN BID12 ) is briefly reviewed in section 2.3. Let the input variable X and the target variable Y distributed according to some joint data distribution p(x, y). The goal of the IB BID25 BID5 is to obtain a compressive representation Z from the input variable X, while maintaining the predictive information about the target variable Y as much as possible. The objective for the IB is DISPLAYFORM0 where I(·, ·) denotes MI and β ≥ 0 is a Lagrange multiplier. The goal is to obtain the optimal representation encoder q φ (z|x) that balances the trade-off between the maximization and minimization of both MI terms. Hence, the IB objective in Eq. provides a natural means for good representations by enforcing the representation Z to ignore irrelevant information from the input and simultaneously to be predictive about the target, which can act as a minimal sufficient statistic of X for predicting Y BID25 BID5.A growing body of studies BID5 BID1 BID13 supports that the learned representations adapting the IB objective tend to be highly efficient and distilled in terms of its code length ). As a consequence, the learned representation is more generalizable and robust to adversarial attack BID5, disentangled BID10 and invariant to nuance factors BID0. Moreover, the IB framework prevents weight over-fitting BID1 BID27, and can be used to visualize high dimensional embedding in a low dimensional latent space . β-VAE BID20 ) is one of the state-of-the-art unsupervised disentangled representation learning models. The key idea of β-VAE is to multiply a constant β ≥ 1 to the KL-divergence term of the original VAE's objective : DISPLAYFORM0 where the encoder q φ (z|x) is generally known as the variational approximation to the intractable p(z|x), p(z) is a prior for the latent representation and p θ (x|z) is the decoder in the VAE context. Recently, a notable connection between β-VAE and the IB theory has been discovered in BID5. Eq. can be derived from the variational approximation to the IB objective Eq.. To clarify this connection, see the variational upper and lower bound of the MI: DISPLAYFORM1 The MI in Eq. subscribed with q: DISPLAYFORM2, is called as the representational MI 1. Given that computing marginal q φ (z) is intractable, we can use any prior p(z) to substitute for q φ (z), forming the variational upper-bound in Eq. FORMULA2 2. Likewise, we can use any decoder model p θ (x|z) to approximate q φ (x|z) = q φ (z|x)p(x)/q φ (z) of the MI, forming the variational lower-bound in Eq.. If the target variable Y in Eq. FORMULA0 is replaced with X, the task is to reconstruct (auto-encode) data from the representation Z. The variational lower-bound of Eq. FORMULA0 obtained by leveraging the upper and lower bound of MI in Eq. corresponds to Eq. FORMULA1 3.The disentanglement-promoting behavior of the β-VAE based on IB theory is discussed in BID10. Constraining the MI (or minimizing KL-divergence in practice) forces the encoder to learn representation containing only strongly relevant information to the data reconstruction, while ignoring other unnecessary (or less-necessary) features. The encoder becomes reluctant to use more channels (or dimensions) of the latent vector to lower the MI constraining cost. Hence, the most distinctive and principle features of data are grouped and aligned along with each independent dimension of the representation space. Generative Adversarial Networks (GAN) BID18 ) establish a min-max adversarial game between two neural networks, a generator G and a discriminator D. The discriminator D aims to distinguish well between real sample x ∼ p(x) and synthetic sample created by the G(z) with a random noise z ∼ p(z), while the generator G is trained to produce a realistic sample that is indistinguishable from the true sample. The adversarial game is formulated as follow: DISPLAYFORM0 Under an optimal discriminator D *, Eq. theoretically involves with the Jensen-Shannon divergence between the synthetic and the true sample distribution: JS(G(z)||p(x)). However, Eq. does not have any specific guidance on how G utilizes a mapping from z to x. That is, the variation of z in any independent dimension often yields entangled effects on a generated sample x. On the other hand, InfoGAN BID12 ) is capable of learning disentangled representations. InfoGAN introduces an additional latent code c and encourages it to describe the semantic features of the data. To do so, the training objective of InfoGAN accommodates a mutual information maximization term between the latent code c and the generated sample x = G(z, c): DISPLAYFORM1 where I(·, ·) denote MI and λ is a weight coefficient. To optimize Eq., the variational lower bound of MI is also exploited similar to that of the IM algorithm BID8. 1 We distinguish it from the generative in the next section. 2 The variational inference relies on the positivity of the KL divergence: BID25 BID28. DISPLAYFORM2 3 A constant data entropy term DISPLAYFORM3 is ignored for brevity. We introduce IB-GAN for disentangled representation learning approach in section 3.1, and propose a practical variational approximation for IB-GAN model in section 3.2. Finally, we discuss some distinctive characteristics of the IB-GAN in-depth in section 3.3. Although InfoGAN BID12 ) is a fully unsupervised GAN-based approach for learning disentangled representations, its disentanglement performance is, constantly reported, lower than β-VAE and its variants BID20; BID11. we hypothesis the weakness of InfoGAN in independent factor learning may originate from the absence of information constraint or any compression mechanism for the representation. Hence, our motivation is straightforward; we adopt the IB principle to the objective of InfoGAN, presenting Information Bottleneck GAN (IB-GAN). IB-GAN not only maximizes the MI term as the original InfoGAN does, but also constrains the maximization of MI simultaneously as DISPLAYFORM0 DISPLAYFORM1 where I L (·, ·) and I U (·, ·) denote the lower and upper bound of generative MI 4 respectively. The parameters λ and β are the weight coefficients of the GAN loss and the upper-bound of MI, respectively. More details on these parameters are discussed in section 3.3. One important change 5 in Eq. compared to the InfoGAN objective is regularizing the upper bound of MI with β, analogously to that of β-VAE and IB theory. For the optimization of IB-GAN, we here define the tractable variational lower and upper bound of the MI in Eq. using the similar derivation in BID12 BID2. For notational consistency, we use p θ (x|z) to denote the generator G(z). Then, the variational lowerbound I L (z, G(z)) of the generative MI in Eq. becomes DISPLAYFORM0 Since the generator marginal p θ (x) is difficult to calculate, a reconstructor model q φ (z|x) is introduced to approximate the quantity p θ (z|x) = p θ (x|z)p(z)/p θ (x) in Eq.. The lower-bound holds thanks to positivity of KL-divergence. Intuitively, by improving the reconstruction of an input code z from a generated sample x = G(z), we can maximize the lower-bound of MI between the generator and the code z. In contrast to the lower-bound, obtaining a practical variational upper-bound of the generative MI is not trivial. If we follow the same approach in BID5, the upper-bound I U (z, G(z)) of the generative MI becomes DISPLAYFORM1 where d(x) is a variational approximation to the generator marginal p θ (x) = z p(x|z)p(z). However, one critical problem of this approach is, in practice, it is difficult to choose or correctly identify the proper approximation model for d(x). Algorithm 1 IB-GAN training algorithm Input: batch size B, hyperparameters λ, β, and learning rates DISPLAYFORM2 DISPLAYFORM3 In theory, we can choose any model for d(x) (e.g. Gaussian), yet any improper choice of d(x) may severely downgrade the quality of synthesized samples from the generator p θ (x|z) since the upperbound I U (z, G(z)) in Eq. FORMULA11 is eventually identical to the KL(p θ (x|z)||d(x)). Moreover, although we express G(z) as p θ (x|z) for notional convenience, the probabilistic modeling of generator G will lose the merit of GAN: the likelihood-free (or implicit) modeling assumption. For this reason, we develop another formulation of the variational upper-bound on the MI term, based on the studies of deep-learning architecture and IB theory BID24 BID0. We define an additional stochastic model e ψ (r|z) that takes a noise input vector z and produces an intermediate stochastic representation r. In other words, we let x = G(r(z)) instead of x = G(z), then we can express the generator as p θ (x|z) = r p θ (x|r)e ψ (r|z). Consequently, a practical upper-bound I U (z, R(z)) of the generative MI can be obtained as: DISPLAYFORM4 DISPLAYFORM5 The first inequality in Eq. holds thanks to the Markov property BID24: if any generative process follows Z → R → X, then I(Z, X) ≤ I(Z, R). The inequality in Eq. FORMULA0 holds from the positivity of KL divergence. Thus, any prior m(r) can be utilized for substituting the marginal e ψ (r) without affecting the generated samples directly; therefore, this can bypass the difficulty of choosing the prior d(x) in Eq..Finally, from the variational lower-bound of the MI in Eq. and the newly introduced upper-bound in Eq., the lower-bound of IB-GAN objective in Eq. can be written as: max DISPLAYFORM6 DISPLAYFORM7 In other words, the intermediate representation r and the KL(e ψ (r|z)||m(r)) with β in Eq. FORMULA0 are leveraged to constrain the amount of shared information between the generator G(z) and input z. Eq. FORMULA0 is optimized by alternatively maximizing the generator G = p θ (x|r), the representation encoder e ψ (r|z), the variational reconstructor q φ (z|x) and the discriminator D. The IB-GAN architecture is presented in FIG0 (a), and overall training procedure is described in Algorithm 1. Connection to rate-distortion theory. Information Bottleneck theory is a generalization of the rate-distortion theory BID25 authors, 2019), in which the rate R is the code length per data sample to be transmitted through a noisy channel, and the distortion D represents the approximation error of reconstructing the input from the source code authors, 2019; ). The goal of RD-theory is minimizing D without exceeding a certain level of rate R, can be formulated as min R,D D + βR, where β ∈ [0, ∞] decides a theoretical achievable optimal frontier in the auto-encoding limit.Likewise, z and r in IB-GAN can be treated as an input and the encoding of the input, respectively. The distortion D is minimized by optimizing the variational reconstructor q φ (z|x(r)) to predict the input z from its encoding r, that is equivalent to maximizing I L (z, G(z)). The minimization of rate R is related minimizing the KL(e ψ (r|z)||m(r)) which measures the in-efficiency (or excess rate) of the representation encoder e ψ (r|z) in terms of how much it deviates from the prior m(r).Disentanglement-promoting behavior. The disentanglement-promoting behavior of β-VAE is encouraged by the variational upper-bound of MI term (i.e. KL(q(z|x)||p(z))). Since p(z) is often a factored Gaussian distribution, the KL-divergence term is decomposed into the form containing a total correlation term (; ; BID11 BID17 BID10, which essentially enforces the encoder to output statistically factored representations (; BID11 . Nevertheless, in IB-GAN, a noise input z is fed into the representation encoder e ψ (r|z) instead of the image x. Therefore, the disentangling mechanism of IB-GAN must be different from those of β-VAEs. From the formulation of the Eq., we could obtain another important insight: the GAN loss in IB-GAN can be seen as the secondary capacity regularizer over the noisy channel since the discriminator of GAN is the JS-divergence (or the reverse KL-divergence) between the generator and the empirical data distribution p(x) in its optimal BID18 BID22. Hence, λ controls the information compression level of z in the its encoding x = G(r(z)) 6. In other words, the GAN loss in IB-GAN is a second rate constraint in addition to the first rate constraint KL(e ψ (r|z)||m(r)) in the context of the rate-distortion theorem. Therefore, we describe the disentanglement-promoting behavior of IB-GAN regarding the ratedistortion theorem. Here, the goal is to deliver the input source z through the noisy channel using the coding r and x. We want to use compact encoding schemes for r and x. The efficient encoding scheme for r is defined by minimizing KL(e ψ (r|z)||m(r)) with the factored Gaussian prior m(r), which promotes statistical independence of the r. The efficient encoding scheme for x is defined by minimizing the divergence between G(z) and the data distribution p(x) via the discriminator; this promote the encoding x to be the realistic image. Maximizing I L (z, G(z)) in IB-GAN indirectly maximize I(r, G(r)) since I(z, G(z)) ≤ I(r, G(r)). In other words, maximizing the lower-bound of MI will increases the statistical dependency between the coding r and G(r), while these encoding need to be efficient in terms of their rate. Therefore, a single independent changes in r must be coordinated with the variations of a independent image factor. How to choose hyperparameters. Although setting any positive values for λ and β is possible, we set β ∈ and fix λ = 1. We observe that, in the most of the cases, IU (r, R(z)) collapses to 0 when β > 0.75 in the experiments with dSprites. Although λ is another interesting hyperparameter that can control the rate of x (i.e. the divergence of the G(z) from p(x)), we aims to support the usefulness of IB-GAN in the disentangled representation learning tasks, and thus we focus on the effect of β ∈ [0, 1.2] on the I U (r, R(z)) while fixing λ = 1. More discussion on the hyperparameter setting will be discussed in Appendix. (; BID16 . Our model's scores are obtained from 32 random seeds, with a peak score of (0.91, 0.78). The baseline scores except InfoGAN are referred to BID17. We use DCGAN with batch normalization as our base model for the generator and the discriminator. We let the reconstructor share the same frontend feature with the discriminator for efficient use of parameters as in the InfoGAN BID12. Also, the MLP-based representation encoder is used before the generator. We train the model using RMSProp BID23 optimizer with momentum of 0.9. The minibatch size is 64 in all experiments. Lastly, we constrain true and synthetic images to be normalized as [−1, 1]. Almost identical architectural configurations for the generator, discriminator, reconstructor, and representation encoder are used in all experiments except that the numbers of parameters are changed depending on the datasets. We defer more details on the models and experimental settings to Appendix. Although it is not easy to evaluate the disentanglement of representations, some quantitative metrics BID20; BID11 BID16 have been proposed based on the synthetic datasets providing ground-truth generative factors such as dSprites or teapots BID16. We verified our approach with the two different metrics (; BID16 on the dSprites dataset since this setting is tested with many other state-of-the-art baselines in BID17 including standard VAE , β-VAE BID20, TC-VAE BID11 and HFVAE BID17.In experiments, we adopt the instance noises technique BID21 since the dSprites images are too simple for the generator of GAN to learn. That is, the intensity distribution of synthetic images is unnaturally narrow (i.e.), making the overlapping probability with generated images very low, where the generator may barely learn from the discriminator. Hence, by adding instance noises ∼ N (0, σ instance * I) to both true and generated inputs, we can significantly improve the training stability of GAN models. This may be the reason for the inconsistency between the previous experiments (; BID11 on InfoGAN and our . For the technical detail, we anneal σ instance linearly from 1 to 0 during training for InfoGAN and IB-GAN. FIG0 shows the variations of KL(e(r i |z)||m(r i)) for 10-dimensional r (i.e. i = 1, . . ., 10) over training iterations on dSprites dataset . The sum of these values is the upper-bound of MI. We observe that all factors of variations are capped by different values. Similar behavior is exhibited in β-VAE BID10. During training, the encoder e ψ (r|z) is slowly adapted to capture the independent factors of dSprites dataset as the lower-bound of MI increases. We present the visual inspection of the latent traversal BID20 with the learned IB-GAN model in FIG2. The IB-GAN successfully learns 5 out of 5 ground truth factors from dSprites dataset, including positions of Y and X, scales, rotations, and shapes, which aligns with the caps on KL scores in FIG0. More and discussion about the convergence and the effects of β will is in Appendix B. TAB0 shows the quantitative in terms of the two disentanglement metric scores (; BID16 . IB-GAN outperforms other baselines (; BID20 BID11 BID17 . Interestingly, in our experiments, InfoGAN attains comparable scores to those of other VAE-based models. On the Eastwood's randomforest metric, InfoGAN slightly outperforms other baselines as well, which is consistent with the of BID16 . Following BID12 BID20 BID11), we evaluate the qualitative of IB-GAN by inspecting latent traversals. As shown in FIG4, the IB-GAN discovers various human attributes such as azimuth, hair color and smiling face expression. In addition, generated images of the IB-GAN are sharp and realistic than the of β-VAE and its variants BID20; BID11. We also show our qualitative on 3D Chairs dataset in FIG4. IB-GAN successfully disentangles scales, leg types and azimuth of chairs. These attributes are hardly captured in the original InfoGAN BID12 BID20; BID11, demonstrating the effectiveness of our approach. The proposed IB-GAN is a novel unsupervised GAN-based model for learning disentangled representation. We made a crucial modification on the InfoGAN's objective inspired by the IB theory and β-VAE; specifically, we developed an information capacity constraining term between the generator and the latent representation. We also derived a new variational approximation technique for optimizing IB-GAN. Our experimental showed that IB-GAN achieved the state-of-the-art performance on disentangled representation learning. The qualitatively generated samples of IB-GAN often had better quality than those of β-VAE on CelebA and 3D Chairs. IB-GAN attained higher quantitative scores than β-VAE and InfoGAN with disentanglement metrics on dSprites dataset. There are many possible directions for future work. First, our model can be naturally extended to adapt a discrete latent representation, as discussed in section 3.3. Second, many extensions of β-VAE have been actively proposed such as BID10; BID11 BID17, most of which are complementary for the IB-GAN objective. Further exploration toward this direction could be another interesting next topic. Reconstruction of input noise z. The ing architecture of IB-GAN is partly analogous to that of β-VAE since both are derived from the IB theory. However, β-VAE often generates blurry output images due to the large β > 1 (; BID11 BID17 since setting β > 1 typically increases the distortion . Recently, demonstrates the possibility of achieving small distortion with the minimum rate by adopting a complex auto-regressive decoder in β-VAE and by setting β < 1. However, their experiment is performed on relatively small dataset (e.g. MNIST, Omniglot).In contrast, IB-GAN may not suffer from this shortcoming since the generator in IB-GAN learns to generate image by minimizing the rate. Moreover, it does not rely on any probabilistic modeling assumption of the decoder unlike VAEs and can inherit all merits of InfoGANs (e.g. producing images of good quality by an implicit decoder, and an adaptation of categorical distribution). One downside of our model would be the introduction of additional capacity control parameter λ. Although, we fixed λ = 1 in all of our experiment, which could also affect the convergence or the generalization ability of the generator. Further investigation on this subject could be an interesting future work. Behaviors of IB-GAN according to β. If β is too large such that the KL-divergence term is almost zero, then there would be no difference between the samples from the representation encoder e ψ (r|z) and the distortion prior m(r). Then, both representation r and generated data x contain no information about z at all, ing in that the signal from the reconstructor is meaningless to the generator. In this case, the IB-GAN reduces to a vanilla GAN with an input r ∼ p(r).Maximization of variational lower-bound. Maximizing the variational lower-bound of generative MI has been employed in IM algorithm BID2 and InfoGAN BID12. Recently, offer the lower-bound of MI, named GILBO, as a data independent measure for the complexity of the learned representations for trained generative models. They discover the optimal lower-bound of the generative MI correlates well with the common image quality metrics of generative models (e.g. or). In this work, we discover a new way of upper-bounding the generative MI based on the causal relationship of deep learning architecture, and show the effectiveness of the upper-bound by measures the disentanglement of learned representation. Implementation of IB-GAN. Since the representation encoder e ψ (r|z) is stochastic, reparametrization trick is needed to backpropagate gradient signals for training the encoder model. The representation r can be embedded along with an extra discrete code c ∼ p(c) before getting into the generator (i.e. G(r, c)), and accordingly the reconstructor network becomes q(r, c|x) to predict the discrete code c as well. In this way, it is straightforward to introduce a discrete representation into IB-GAN, which is not an easy task in β-VAE based models. Theoretically, we can choose the any number for the dimension of r and z. However, The disentangled representation of IB-GAN is learned via the representation encoder e ψ (r|z). To obtain the representation r back from the real data x, we first sample z using the learned reconstructor q φ (z|x), and input it to the representation encoder e ψ (r|z). Therefore, we typically choose a smaller r dimension than that of z. For more details on the architecture of IB-GAN, please refer Appendix. E.Related Work. Many extensions of β-VAE BID20 have been proposed. BID10 modify β-VAE's objective such that the KL term is minimized to a specific target constant C instead of scaling the term using β. and BID11 demonstrate using the ELBO surgery that minimizing the KL-divergence enforces factorization of the marginal encoder, and thus promotes the independence of learned representation. However, a high value of β can decrease the MI term too much, and thus often leads to worse reconstruction fidelity compared to the standard VAE. Hence, they introduce a total correlation BID26 based regularization to overcome the reconstruction and disentanglement trade-off. These approaches could be complementary to IB-GAN, since the objective of IB-GAN also involves with the KL term. This exploration could be an interesting future work. One of the most important hyperparameter in the IB-GAN objective of Eq. FORMULA0 is β that controls the ratio of the lower-bound I L (z, R(z)) and the upper-bound I U (z, R(z)). Hence, the optimal balance point between the lower and upper bound term is affected by the β. Each panel in FIG6 shows the variational lower and upper-bound of MI along with independent KL(e(r i |z)||m(r i)) for each r i (i = 1, · · ·, 10) over the 150K training iterations. As shown in FIG6, if β = 0, the upper-bound of MI in Eq. FORMULA0 is ignored and the constraining effect on the representation r disappears. Hence, the lower-bound of MI can quickly increases up to its natural upper-bound 7 similar to the MI lower-bound in InfoGAN. With β = 1 of FIG6, the upper-bound of MI drops down to almost zero and so does the lower-bound. Hence, the representation r is independent of z (i.e. dose not contain any information about z) and IB-GAN reduces to vanilla GAN.When β is set properly as in FIG6, both lower and upper-bound of MI increase smoothly and the representation encoder e ψ (r|z) is slowly adapted to capture the distinctive factors of the dataset, where independent KL-divergence KL(e(r i |z)||m(r i)) increases one by one by capturing each disentangled attribute. Note that the sum of individual KL scores is the upper-bound of MI I U (z, R(z)) = i KL(e(r i |z)||m(r i)). We observe that all factors of variations are capped by different values; this behavior is reported as a key element of the disentangled representation learning in β-VAE BID10. FORMULA0 ) vs β. FIG7 and 5b illustrates the expected converged value of upper and lower MI bounds over the different β. Overall, the upper MI bound tends to decrease exponentially as β increases, consequently the lower MI bound decreases as well. DISPLAYFORM0 Specifically, β = 0, the upper-bound MI term disappears in the IB-GAN Eq.. Hence, the representation encoding r can diverge from the prior distribution m(r) without any restrictions, ing in a high value of upper MI bound. Interestingly, the gap between the upper and lower bound is also reduced as the β parameter increases as we can see in Figure. 5b. Lastly, FIG7 shows the effect of β on the disentanglement scores. The optimal disentanglement score was achieved when the β is around in a range of [0.1, 0.35], and optimal disentanglement score 0.91 is obtained when β = 0.212 supporting the fact that IB-GAN could control the disentanglement of the learned representation with the upper-bound of generative MI and the varying β. Following BID12 BID20 BID11 ), we evaluate the qualitative of IB-GAN by inspecting latent traversals. As shown in Figure7, the IB-GAN discovers various human attributes such as (a) azimuth, (b) color, (c) hair color, (d) skin color, (e) smile, and (f) gender. All of the features in Figure7 and Figure3(a) are captured by one best model with the parameter of β = 0.2838, λ = 1. These attributes are hardly captured in the original InfoGAN BID12 BID20; BID11, demonstrating the usefulness of the upper-bound of generative MI in IB-GAN. In addition, generated images of the IB-GAN are often sharp and realistic than the of β-VAE and its variants BID20; BID11. BID20 737,280 binary 64 × 64 images of 2D shapes with 5 ground truth factors. Ground truth factors consist of 3 shapes, 6 scales, 40 orientations, and 32 positions of X and Y. 3D Chairs BID6 86,366 gray-scale 64 × 64 images of 1,393 chair CAD models with 31 azimuth angles and 2 elevation angles. CelebA 202,599 RGB 64 × 64 × 3 images of celebrity faces consisting of 10,177 identities, 5 landmark locations, and 40 binary attributes. We use the cropped version of the dataset. TAB2 describes the details of hyperparameter settings used in our experiments. LR G/E/Q 5e-5, D 1e-6 iterations 1.5e5 RMSProp(momentum=0.9), nc=1, ngf=32, ndf=16, z dim=64, r dim=10, λ=1, β=0.2Instance Noise: σ instance is annealed linearly from 1.0 to 0 for 1.3e5 iterations. LR G/E/Q 5e-5, D 5e-6 iterations 2e5CelebA RMSProp(momentum=0.9), nc=3, ngf=64, ndf=64, z dim=500, r dim=15, λ=1, β=0.2838LR G/E/Q 5e-5, D 5e-7 iterations 2.5e5We summarize some implementation details of the models in our experiments on dSprites dataset, 3D Chairs, and CelebA datasets. TAB3 shows the base architectures of IB-GAN for the generator, discriminator, and encoder, while TAB4 shows those of InfoGAN. TAB2 also presents the hyperparameter settings that we use for the models in all experiments. TAB2 for hyperparameter setting. Generator ( DISPLAYFORM0

We introduced a novel, simple, and efficient data augmentation method that boosts the performances of existing GANs when training data is limited and diverse.

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.


Question: Represent the article for finding a single-sentence TLDR

Question: We develop a stochastic whole-brain and body simulator of the nematode roundworm Caenorhabditis elegans (C. elegans) and show that it is sufficiently regularizing to allow imputation of latent membrane potentials from partial calcium fluorescence imaging observations. This is the first attempt we know of to ``complete the circle,'' where an anatomically grounded whole-connectome simulator is used to impute a time-varying ``brain'' state at single-cell fidelity from covariates that are measurable in practice. Using state of the art Bayesian machine learning methods to condition on readily obtainable data, our method paves the way for neuroscientists to recover interpretable connectome-wide state representations, automatically estimate physiologically relevant parameter values from data, and perform simulations investigating intelligent lifeforms in silico. One of the goals of artificial intelligence, neuroscience and connectomics is to understand how sentience emerges from the interactions of the atomic units of the brain, to be able to probe these mechanisms on the deepest level in living organisms, and to be able to simulate this interaction ad infinitum. In this work, we assemble an anatomically grounded, interpretable probabilistic brainbody simulator for the widely studied nematode roundworm Caenorhabditis elegans (C. elegans). We then present methods for performing posterior inference in the time evolution of the state of the worm and estimate the global simulator parameter values from readily obtainable non-invasive calcium fluorescence data. We refer to using an anatomically grounded model to infer latent states and parameters, conditioned on partial data, as a "virtual patch clamp" (VPC). The VPC also facilitates in silico experimentation on "digital" C. elegans specimens, by programmatically modifying the simulator and observing the ing simulations; enabling rapid, wide-reaching, fully observable and perfectly repeatable exploration of hypotheses into the how the fundamental units of the neural circuit of C. elegans combine to create intelligent behaviour. Due to the simplicity and regularity of its anatomy, and its predictable yet sophisticated behavioural repertoire, C. elegans is used as a "model organism" across biology and neuroscience research. Notably, its connectome is regular across wild-type specimens and has been mapped at synapse and. (e) gap junction fidelity using electron microscopy. Because of this fixed architecture, neural circuit simulators, imbued with anatomically correct structure, have been developed to produce feasible whole C. elegans connectome simulators by leveraging highly accurate neural dynamics models. Likewise, its simple anatomy has allowed body and locomotion simulators to be developed. The first contribution of this paper is a new C. elegans simulator that integrates existing simulators and models developed by the C. elegans community. At a high level, our simulator is comprised of three components: a simulator for the time-evolution of the membrane potential and intracellular calcium ion concentration in all 302 C. elegans neurons, a simulator for the physical form of the worm and the associated neural stimuli and proprioceptive feedback, and a model relating the intracellular calcium to the observable florescence data. The first component of our model is a simulator of connectome-scale, single-neuron fidelity neural dynamics. We modify the simulator presented by Marblestone, which builds on Wicks et al., called'simple C. elegans' (SCE). SCE is designed to be an easily interpretable simulator of C. elegans neural membrane potential dynamics via single-compartment neuron models connected by chemical synapses and electrical gap junctions. Exemplar voltage traces generated by our simulator are shown as black dashed lines in Figure 1 (b). We add to SCE a model for intracellular calcium ion concentration. We also integrate a simulator of the body shape of the worm, WormSim. WormSim models the body shape in two dimensions as a series of rods, contractile units and springs driven by impulses generated by a simplified neural network. We integrate the anatomically correct representation used by SCE to drive WormSim and receive proprioceptive feedback. A typical evolution of body state is shown in black in Figure 1 (e). Finally we incorporate a model of the fluorescence signals observed through calcium imaging. This dependence is described by a saturating Hill-type conditioned on intracellular calcium concentration, where only M of the 302 neurons are observed and identified (here M = 49, see Kato et al. ). To summarize our model, the neuron states, body state, and proprioceptive feedback define the latent "brain" and "body" state of the worm, denoted at time t as x t ∈ R 994. The observed data, y t ∈ R M +, is the calcium florescence signal. We now demonstrate how the tools of Bayesian inference can be employed to condition simulations on partial observations, make predictions conditionally or unconditionally, and perform marginal maximum a posteriori parameter estimation. The second contribution of this paper is the adoption and scaling of a method to impute the entire latent state, x t, conditioned on observable calcium imaging florescences. We wish to quantify the distribution over the latent states conditioned on the observed data, referred to as the posterior distribution p(x 0:T |y 1:T, θ). To relate how this achieves to our outlined objectives, this represents, under the model, the distribution over all latent neural and physiological states, x t, conditioned on the observed data, providing the imputation element of the VPC. Forward simulation of the particles initialized from the posterior distribution at T provides posterior predictive inference over state evolution, where, for instance, physiological variables can be programmatically clamped (inspiring the name VPC). Finally the posterior, p(θ|y 1:T) = p(y 1:T |θ)p(θ), allows us to objectively compare models and hypotheses, which will be used later for parameter estimation. Due to the non-invertible, non-differentiable nature of the simulator, we use sequential Monte Carlo (SMC) for estimating the posterior as a weighted discrete measure approximating the target distribution, as well as providing an estimation of the model evidence, p(y 1:T |θ). In our first experiment we first generate a synthetic state trajectory by sampling from the model, and then recover the known ground-truth trajectory from observed fluorescence traces using a fixed model. Specifically we condition on the same 49 neurons identified in the calcium imaging data released by Kato et al.. Results for this are shown in Figure 1(b), where the true state is shown in black, while the filtering distribution recovered by SMC is shown in blue. The blue reconstructions are congruent with the black trace, indicating that the latent behaviour of the complete system is being well-reconstructed despite partial observability. Critically, neurons not directly connected to observed neurons (for instance VD6) are correctly reconstructed, indicating that the regularizing capacity of the model is sufficient to constrain these variables. Further confirmation of the power of this method can be seen in the leftmost column of Figure 1(e), showing the predicted body shape closely matches the true state, despite not being explicitly conditioned upon body shape. This experiment shows that the VPC is tractable and is capable of yielding high-fidelity reconstructions of pertinent latent states given partial calcium imaging observations via the application of Bayesian inference to time series models of C. elegans. The posterior inference and evidence approximation presented in the previous section is useful for imputing values and performing in silico experimentation. In the previous section we fixed the model to demonstrate the viability of SMC for latent state imputation. We now allow the parameters of the simulator, collectively denoted θ, such as the non-directly observable electrical and chemical characteristics of individual synapses in the C. elegans connectome, as well as parameters of the body model, the calcium fluorescence model, etc, to be unknown and hence must be learned. We conclude this paper by taking concrete steps towards performing such parameter estimation, as defined by the simulator-structured hypothesis class defined by the chosen model. Our goal is to estimate the best simulator parameters θ * given observed data, i.e. θ * = argmax θ p(θ|y) = argmax θ p(y|θ)p(θ). The method we employ for performing parameter estimation is a novel combination of variational optimization (VO) and SMC evidence approximation. This in a stochastic gradient for parameter estimation that does not require a differentiable simulator and can deal with a large number of latent variables. VO starts with the following bound The gradient of U (φ) with respect to φ can then be computed as where Monte Carlo integration is used to evaluate this expectation. The objective function is the joint density f (θ) = −p(y, θ) = −p(y|θ)p(θ), where the likelihood term is approximated via SMC. To our knowledge, this is the first time that pseudo-marginal methods have been paired with variational optimization methods. We refer to this procedure as particle marginal variational optimization (PMVO). We implement a framework for embarrassingly parallel evaluation of multiple SMC sweeps on large, distributed high performance compute clusters, where each SMC sweep is executed on a single node, eliminating network overheads. We conclude by demonstrating the utility of our PMVO technique by recovering known simulator parameters on synthetic data generated by the model. For this work, we optimize the two parameters we introduced by integrating SCE and WormSim, namely the strength of motor stimulation, w m, and proprioceptive feedback, w s. The of this experiment are shown in Figure 1. Figures 1(c) and 1(e) show the imputed voltage traces and body poses when using the true parameters (blue), initial parameters (red) and optimized parameters (green), conditioned on just 49 neurons. Recovery of "good" parameter values facilitates good imputation of latent states, especially for body position which is not explicitly conditioned on after initialization. Figure 1(d) shows the distribution of convergence of the two parameters towards the true value. This experiment shows that parameter inference in C. elegans models using PMVO is viable. Increasing the number of particles used in the SMC sweeps, the number of samples drawn from the proposal and observing more neurons (although currently logistically infeasible) improves the quality of the reconstructions and recovery of parameters. In this work we have explored performing Bayesian inference in whole-connectome neural and whole-body C. elegans simulations. We describe the model-based Bayesian inference aspect of this as a "virtual patch clamp," whereby unobserved latent membrane potentials can be inferred from partial observations gathered non-invasively. Our choice of inference method facilitates estimation of the model evidence, a measure of how well the model explains the observed data. We presented a method for maximizing this evidence without requiring differentiable simulation components. In the past year several articles discussing open research issues pertaining to C. elegans simulation have been produced by the C. elegans community. Figure 1 (a) outlines the community planned development pipeline for C. elegans simulation. Our work addresses the implementation of the box simply labelled "optimization." We show on representative synthetic data that our method is capable of performing such an optimization. This approach promises to allow neuroscientists to peer deeper into the neural function of a living organism, testing hypothesis on neural function that were previously unreachable. It is widely touted that convolutional neural networks were developed by wide-scale study of the V1 cortex. We believe connectome-level optimization and simulation, as demonstrated here, is the next step in neuroscience to understanding the very root of intelligence, but also discovering and developing techniques building towards artificial general intelligence.

Represent the short summary for finding the corresponding scientific article (long)

We develop a whole-connectome and body simulator for C. elegans and demonstrate joint state-space and parameter inference in the simulator.

Represent the TLDR for finding the corresponding scientific article

We use continuous time dynamics to define a generative model with exact likelihoods and efficient sampling that is parameterized by unrestricted neural networks.


Question: Represent the article for finding a single-sentence TLDR

Question: The high-quality node embeddings learned from the Graph Neural Networks (GNNs) have been applied to a wide range of node-based applications and some of them have achieved state-of-the-art (SOTA) performance. However, when applying node embeddings learned from GNNs to generate graph embeddings, the scalar node representation may not suffice to preserve the node/graph properties efficiently, ing in sub-optimal graph embeddings. Inspired by the Capsule Neural Network (CapsNet), we propose the Capsule Graph Neural Network (CapsGNN), which adopts the concept of capsules to address the weakness in existing GNN-based graph embeddings algorithms. By extracting node features in the form of capsules, routing mechanism can be utilized to capture important information at the graph level. As a , our model generates multiple embeddings for each graph to capture graph properties from different aspects. The attention module incorporated in CapsGNN is used to tackle graphs with various sizes which also enables the model to focus on critical parts of the graphs. Our extensive evaluations with 10 graph-structured datasets demonstrate that CapsGNN has a powerful mechanism that operates to capture macroscopic properties of the whole graph by data-driven. It outperforms other SOTA techniques on several graph classification tasks, by virtue of the new instrument. GNN is a general type of deep-learning architectures that can be directly applied to structured data. These architectures are mainly generalized from other well-established deep-learning models like CNN BID9 and RNN BID12. In this paper, we mainly focus on Convolution-based Graph Neural Networks which attract increasing interest recently. Convolution operation can be embedded into Graph Neural Networks from spectral or spatial perspective. BID1 defines the convolution operation in the Fourier domain which needs to calculate the eigendecomposition of the graph Laplacian. This method is computationally expensive and the filters they defined are non-spatially localized. Later, BID4 introduces Chebyshev expansion of the graph Laplacian to avoid computing eigenvectors and BID8 proposes to do convolution within 1-step neighbor nodes to reduce the complexity. From the spatial perspective, BID3 and propose to define a node receptive-field and do convolution within this field during which the information of each node as well as their neighbor nodes is gathered and new representation of each node is generated through an activation function. Both of these two perspectives perform well in node representation learning and a number of variants BID20 are developed based on the convolution idea and some of them have proven to achieve SOTA in various tasks. The success of GNN in node representation learning has inspired many deep-learning-based approaches to leverage on node embeddings extracted from GNN to generate graph embeddings for graph-based applications. However, during this procedure, the learned representation of each node will be considered as multiple individual scalar features instead of one vector. For example, applies element-wise max-pooling to nodes embeddings when generating graph embeddings, BID22 generates graph embeddings by computing the element-wise covariance of all nodes. These operations indicate that the authors capture node features in the form of scalar when they generate graph embeddings which may not suffice to preserve the node/graph properties efficiently. To build high-quality graph embeddings, it is important to not only detect the presence of different structures around each node but also preserve their detailed properties such as position, direction, connection, etc. However, encoding these properties information in the form of scalar means activating elements in a vector one-by-one which is exponentially less efficient than encoding them with distributed representations. This has been identified discussed in BID16. Inspired by CapsNet, we propose to extend scalar to vector during the procedure of applying GNN to graph representation learning. Compared with scalar-based neural network, vector-based neural network preserves the information of node/graph properties more efficiently. The technique for extracting features in the form of vectors is proposed in BID5 and improved in BID16 and BID6. This technique is mainly devised for image processing. In their work, the extracted vector is referred to as capsule (a group of neurons in neural network), so we follow the same notation in our work. Introducing capsules allows us to use routing mechanism to generate high-level features which we believe is a more efficient way for features encoding. Compared with max-pooling in CNN in which all information will be dropped except for the most active one, routing preserves all the information from low-level capsules and routes them to the closest high-level capsules. Besides, this allows to model each graph with multiple embeddings and each embedding reflects different properties of the graph. This is more representative than only one embedding used in other scalar-based approaches. In this paper, we propose Capsule Graph Neural Network (CapsGNN), a novel deep learning architecture, which is inspired by CapsNet and uses node features extracted from GNN to generate high-quality graph embeddings. In this architecture, each graph is represented as multiple embeddings and each embedding reflects the graph properties from different aspects. More specifically, basic node features are extracted in the form of capsules through GNN and routing mechanism is applied to generate high-level graph capsules as well as class capsules. In the procedure of generating graph capsules, an Attention Module can be applied to tackle graphs in various sizes. It also assigns different weights to each capsule of each node so that this model focuses on critical parts of the graph. We validate the performance of generated graph embeddings on classification task over 5 biological datasets and 5 social datasets. CapsGNN achieves SOTA performance on 6 out of 10 benchmark datasets and comparable on the rest. T-SNE BID11 ) is used to visualize the learned graph embeddings and the show that different graph capsules indeed capture different information of the graphs. Here, we provide a brief introduction to Graph Convolutional Networks (GCNs) BID8, routing mechanism in CapsNet and Attention mechanism which is used in CapsGNN. By definition, a weighted directed graph can be represented by G = (V, X, A) where V = {v 1, v 2, ...v N} is the set of nodes and A ∈ {0, 1} N ×N is the adjacency matrix. If there is an edge from v i to v j, then A ij = 1 otherwise A ij = 0. X ∈ R N ×d represents the features of each node. d is the number of feature channels and N is the number of nodes. GCN, a widely used GNN architecture, is chosen as one of the key building blocks in our work. At each layer of the GCN, the convolution operation is applied to each node as well as its neighbors and the new representation of each node is computed through an activation function. This procedure can be written as: DISPLAYFORM0 where Z l ∈ R N ×d represents nodes features at the layer l, d represents the number of feature channels and Z 0 = X, W l ∈ R d×d is a trainable weights matrix which serves as a channel filter, f is a nonlinear activation function, T ∈ R N ×N is the information transform matrix and it is usually calculated from the adjacency matrix A for guiding the information flowing between nodes. A complete GNN usually stacks L layers to generate final nodes embeddings Z L. In the architecture proposed by BID8, at the lth layer of GCN, the extracted features of each node actually take all its adjacent nodes within l steps into consideration. So l can be considered as the size of the node receptive-field at this layer. This special property inspired us to use nodes features extracted from different layers to generate the graph capsules. The concept of capsules is invented by Hinton's team BID5 and used recently in BID16 and BID6. CapsNet is designed for image features extraction and it is developed based on CNN. However, unlike traditional CNN in which the presence of feature is represented with scalar value in feature maps, the features in CapsNet are represented with capsules (vectors). In BID16, the direction of capsules reflects the detailed properties of the features and the length of capsules reflects the probability of the presence of different features. The transmission of information between layers follows Dynamic Routing mechanism. The specific procedure of Dynamic Routing can be found in Appendix A for the completeness. Inspired by CapsNet, the capsule mechanism is adopted and fused with GNN in our proposed Caps-GNN to generate graph capsules and class capsules on the basis of node capsules which are extracted from GNN. Dynamic Routing is applied to update weights between capsules from one layer to the next layer so that the properties captured by node capsules can be propagated to suitable graph capsules. Thus, each graph is modeled as multiple graph capsules, and then modeled as multiple class capsules. Different graph capsules reflect the properties of the graph from different aspects. Attention mechanism is widely applied in image BID26 and natural language processing domain BID2 where it is used to find the relevant parts of the input data to the task target. The main procedure of Attention mechanism is: 1) defining an attention measure which is used to measure the relevance of each part of the input data to the task target. 2) normalizing the generated attention value. 3) scaling each part with the normalized attention value. In CapsGNN, we apply Attention mechanism for two purposes: 1) scaling each node capsule so that the graph capsules that are generated from different graphs are still comparable even though these graphs are vastly different in sizes. 2) guiding the model to focus on more relevant parts of graphs. In this section, we outline CapsGNN and show how it is used to generate high-quality graph capsules which then can be applied to graph classification task. Figure 1 shows a simplified version of CapsGNN. It consists of three key blocks: 1) Basic node capsules extraction block: GNN is applied to extract local vertices features with different receptive-field and then primary node capsules are built in this block. 2) High level graph capsules extraction block: Attention Module and Dynamic Routing are fused to generate multiple capsules for graphs. 3) Graph classification block: Dynamic Routing is applied again to generate class capsules for graph classification. The details of each block is explained in the following. Firstly, the basic node features are extracted with GNN. Node degrees can be used as node attributes if nodes do not have attributes. We use the architecture improved by BID8 (GCN) as the node features extractor. The difference is that we extract multi-scale node features from different layers and the extracted features are represented in the form of capsules. The procedure can be written as: DISPLAYFORM0 Figure 1: Framework of CapsGNN. At first, GNN is used to extract node embeddings and form primary capsules. Attention module is used to scale node embeddings which is followed by Dynamic Routing to generate graph capsules. At the last stage, Dynamic Routing is applied again to perform graph classification.where W l ij ∈ R d×d is the trainable weights matrix. It serves as the channel filters from the ith channel at the lth layer to the jth channel at the (l + 1)th layer. Here, we choose f (·) = tanh(·) as the activation function. DISPLAYFORM1 To preserve features of sub-components with different sizes, we use nodes features extracted from all GNN layers to generate high-level capsules. After getting local node capsules, global routing mechanism is applied to generate graph capsules. The input of this block contains N sets of node capsules, each set is S n = {s 11, .., DISPLAYFORM0, where C l is the number of channels at the lth layer of GNN, d is the dimension of each capsule. The output of this block is a set of graph capsules H ∈ R P ×d . Each of the capsules reflects the properties of the graph from different aspects. The length of these capsules reflects the probability of the presence of these properties and the angle reflects the details of the graph properties. Before generating graph capsules with node capsules, an Attention Module is introduced to scale node capsules. Attention Module. In CapsGNN, primary capsules are extracted based on each node which means the number of primary capsules depends on the size of input graphs. In this case, if the routing mechanism is directly applied, the value of the generated high-level capsules will highly depend on the number of primary capsules (graph size) which is not the ideal case. Hence, an Attention Module is introduced to combat this issue. The attention measure we choose is a two-layer fully connected neural network F attn (·). The number of input units of F attn (·) is d × C all where C all = l C l and the number of output units equals to C all. We apply node-based normalization to generate attention value in each channel and then scale the original node capsules. The details of Attention Module is shown in FIG0 and the procedure can be written as: DISPLAYFORM1 wheres n ∈ R 1×C all d is obtained by concatenating all capsules of the node n. DISPLAYFORM2 represents the ith capsule of the node n and F attn (s n) ∈ R 1×C all is the generated attention value. In this way, the generated graph capsules can be independent to the size of graphs and the architecture will focus on more important parts of the input graph. The structure of Attention Module. We first flatten primary capsules and apply two layer fully-connected neural network to generate attention value for each capsule. Node-based normalization (normalize each row here) is applied to generate final attention value. Scaled capsules are calculated by multiplying the normalized value with primary capsules. After Attention Module, coordinate addition module can be used to preserve the position information of each node during the procedure of generating node capsule votes. Here, we introduce coordinate addition as an additional module and more details can be found in Appendix C.The procedure of generating multiple graph capsules is summarized as follows: 1) Scale primary capsules: Apply Attention Module to scale primary capsules. The of this module should be S ∈ R N ×C all ×d.2) Calculate votes: When calculating votes, capsules of different nodes from the same channel share the transform matrix. The of this step is a set of votes V ∈ R N ×C all ×P ×d where C all denotes the number of channels. P denotes the defined number of graph capsules.3) Dynamic Routing Mechanism: High-level graph capsules are computed with the procedure introduced in Section 2.3 based on votes produced in previous steps. This block is designed for graph classification using the graph capsules. Classification Loss. Dynamic Routing is applied again over graph capsules to generate final class capsules C ∈ R K×d, where K is the number of graph classes. Here, we use margin loss function proposed in BID16 to calculate the classification loss and it is computed as: DISPLAYFORM0 where m + = 0.9, m − = 0.1 and T k = 1 iff the input graph belongs to class k. λ is used to stop initial learning from reducing the length of all class capsules especially when K is large. Reconstruction Loss. Following BID16, we use reconstruction loss as regularization method. Here, all class capsules are masked except the correct one and it is decoded with two fullyconnected layer to reconstruct the input information. The information we reconstruct here is the histogram of input nodes. The procedure can be written as: DISPLAYFORM1 where m i represents the number of nodes with the attribute i appear in the input graph, d i is the corresponding decoded value. M P i = 1 iff input graph contains nodes with attribute i. Equation 5 is used to prevent reducing reconstruction loss from setting all decoded value as 0 especially when most of the elements of the ground truth are 0.The architecture details presented in section 3 describe the key design idea of CapsGNN which is based on the fusing of GNN and CapsNet. We also present a general comparison between CapsGNN with existing approaches in Appendix D. We verify the performance of the graph embeddings extracted from CapsGNN against a number of SOTA approaches and some classical approaches on classification task with 10 benchmark datasets. Besides, we conduct experimental study to assess the impact of capsules in efficiency of encoding features of graphs. We also conduct brief analysis on the generated graph/class capsules. The experimental and analysis is shown in the following. In addition to the analysis of the whole framework, we also provide a comparison experiment to evaluate the contribution of each module of CapsGNN with classification task. More details can be found in Appendix F. The goal of graph classification is to predict the classes these graphs belong to by analyzing the structure and nodes labels information of graphs. More specifically, given a set of labeled graphs DISPLAYFORM0 The objective of graph classification is to find a mapping f such that f: G → Y. We compare CapsGNN with both kernel-based and deep-learning-based algorithms. The details are given as follows:Kernel-based Methods: Three kernel-based algorithms, namely the Weisfeiler-Lehman subtree kernel (WL) BID18, the graphlet count kernel(GK) BID17, and the Random Walk (RW) BID23. Typically, kernel-based algorithms first decompose graphs into sub-components based on the kernel definition, then build graph embeddings in a feature-based manner. Lastly, some machine learning algorithms (i.e., SVM) are applied to perform graph classification. Deep-Learning-based Methods: Three types of deep-learning-based algorithms are selected: 1) Graph2vec BID14, Deep Graph Kernel (DGK) BID24 and AWE BID7. Graph2vec, DGK and AWE require extracting substructures in advance while Graph2vec and AWE learn the representations of graphs in the manner of Doc2vec BID10, DGK applies Word2vec BID13 to learn the similarity between each pair of sub-structures which will be used to build the graph kernel. Then kernel-based machine learning methods (i.e., SVM) are applied to perform graph classification. These three algorithms as well as kernel-based methods are all sub-components based and they all require two stages to do graph classification. So although Graph2vec, DGK and AWE apply learning approaches to learn the embeddings, we still consider them and other kernel-based algorithms as the same type in our experiments and we mainly compare our proposed architecture with the other remained methods which are all end-to-end and totally data-driven architectures.2) PATCHY-SAN (PSCN) BID15. This method first sorts all nodes, then defines a receptive-field size for each node. These receptive-field are then filled with sorted neighbor nodes. Lastly, 1-D CNN is applied to perform graph classification.3) GCAPS-CNN BID22, Dynamic Edge CNN (ECC) BID19 and Deep Graph CNN (DGCNN). These methods are all GNN-based algorithms. GCAPS-CNN first extract FGSD BID21 features for nodes that do not have attributes and then generate capsules for each node with higher-order statistical moment value of its neighbor nodes. At the last layer, they calculate covariance between all nodes to generate graph embeddings. ECC extracts node features on the condition of edge labels in GNN and then apply multi-scale pyramid structure to coarsen the graph. It uses average pooling at the last layer to generate graph embeddings. DGCNN generates nodes embeddings through a multi-layer GNN and combine features extracted from all layers. Then they order the nodes based on the embeddings extracted from the last layer which is followed by 1-D CNN. Five biological graph datasets: MUTAG, ENZYMES, NCI1, PROTEINS, D&D and five social network datasets: COLLAB, IMDB-B, IMDB-M, RE-M5K, RE-M12K BID24 are used for our experimental study. Details of these datasets can be found in Appendix B.We applied 10-fold cross validation to evaluate the performance objectively. Each time we use 1 training fold as validation fold to adjust hyper-parameters, 8 training fold to train the architecture and the remained 1 testing fold to test the performance. We stop training when the performance on the validation fold reaches to the highest. Then we use the accuracy on the test fold as our test . The final is the average of these 10 test accuracy. By default, we use the reported in the original work for baseline comparison. However, in cases where the are not available, we use the best testing reported in BID22, and BID7. More details about experimental setting can be found in Appendix E. CapsGNN achieves the SOTA performance on social datasets. More specifically, we are able to improve the classification accuracy by a margin of 2.78% and 5.30% on RE-M5K and RE-M12K respectively. This demonstrates that learning features in the form of capsules and modeling a graph to multiple embeddings is beneficial to capture macroscopic properties of graphs which are more important in classifying social networks. These also consistent with the property of CapsNet, as it focuses more on extracting important information from children capsules by voting. However, applying routing to the whole graph leads to preserve all the information at a graph level and this property is not suitable to give prominence to individual fine structures which might be more important to biological datasets analysis. This in less robust of CapsGNN on biological datasets. Despite this, the performance of CapsGNN in graph classification task still demonstrates its capability of graph representation especially its high potential of large graph dataset analysis. The main objective of this experiment is to examine the efficiency of capsules in encoding graph features. More efficient in feature encoding here means representing more information with the similar number of neurons. We construct a scalar-based neural network for each CapsGNN and then compare the CapsGNN with its related scalar-based architecture by comparing their training and testing accuracy on graph classification task to demonstrate the efficiency in feature representation. More specifically,these scalar-based architectures are designed by replacing the graph capsules block and the class capsules block in CapsGNN with fully-connected layers (FC). In this case, the only difference between each pair of CapsGNN and its corresponding scalar-based architecture is that CapsGNN represents features with vectors and uses routing to propagate information between layers while the scalar-based architecture encodes features with scalar values. In this experiment, the number of layers of GNN is set as L = 3, the number of channels at each layer is all set as C l = 2. We construct different CapsGNNs by adjusting the dimension of nodes (d n) and graphs (d g) capsules and the number of graph capsules (P). The size of FC in scalar-based architectures is adjusted based on the size of CapsGNNs so that they have comparable number of trainable weights. Other hyper-parameters are the same as Appendix E. The details of the tested architectures are shown in TAB3. Besides, NCI1 dataset, which has more than 4000 graphs, is used for the test. The accuracy of NCI1 on various architectures can be found in FIG2. In TAB3 and FIG2, the setting of different architectures is represented as d n -d g -P. Here, we choose the simplest setting as an example: 2-4-2 means that the dimension of nodes capsules is d n = 2, the dimension of graph and class capsules is d g = 4 and the number of graph capsules P equals to 2. Besides, we set the dimension of FC of its corresponding scalar-based architecture as 12 so that they have comparable number of trainable weights. In this case, each graph is modeled as 2 4-dimensional graph embeddings in the CapsGNN or 1 12-dimensional graph embedding in its corresponding scalar-based architecture. Both architectures are sub-optimal to represent the whole dataset while CapsGNN can still reach higher accuracy compared with the scalar-based architecture. As we can see from Figure dimension of FC is slightly higher than the dimension of graph embeddings in CapsGNN, CapsGNN can still reach higher accuracy which indicates that CapsGNN is more powerful in representing the whole dataset. When we keep increasing the number of graph capsules in CapsGNN and enlarging the dimension of FC in scalar-based architectures, the difference between the dimension of graph embeddings and the size of FC becomes larger, their training accuracy will be closer. It is noted that the training accuracy of scalar-based architectures is slightly higher than the CapsGNNs when the dimension of FC is about 20% larger than the dimension of graph capsules. In this experiment, we use extremely simple architectures on purpose to simulate the situation where we need to model complex datasets with relatively simple architectures. Since each pair of CapsGNN and its corresponding scalar-based architecture have similar structure and comparable number of trainable weights, the higher training accuracy and testing accuracy of CapsGNN demonstrate its efficiency in feature encoding and its strong capability of generalization. CapsGNN leverages on capsules idea to get multiple embeddings for each graph so that complex information underlying graphs can be captured more effectively. To explore the properties of the extracted graph/class capsules, we plot the graph distribution based on capsules extracted from different channels with t-SNE. Due to space constrain, we only take REDDIT-M12K as an example. We choose to depict the distribution of graphs which are generated from 3 categories, namely atheism, IAmA and mildlyinteresting with capsules extracted from the 1st, 2nd, 11th, 14th channel of graph capsules. As we can see from TAB4, different channels of capsules represent different aspects of graph properties. atheism and IAmA can be discriminated obviously with capsules extracted from the 11th and the 14th channels while they are hard to be separated with capsules extracted from the 1st and the 2nd channels. However, atheism and mildlyinteresting can be discriminated with the capsules extracted from the 1st and the 2nd channels while they are mixed in the 11th and the 14th channels which is opposite to the case of atheism and IAmA. This phenomenon can also be observed in other multi-class datasets. It is still hard to figure out the specific aspects these capsules focus on. However, compared with scalar-based neural networks, modeling an object with multiple embeddings makes it possible to explore the meaning of each channel which may lead the model to learn more interpretable embeddings in the future. DISPLAYFORM0 As we can see from TAB5, different class capsules focus on different classification-related graph properties. For example, the capsules that represent athesism (first column) can well discriminate athesism (red) from the other two types of graphs while IAmA (green) and mildlyinteresting (blue) are mixed in this channel. The similar phenomenon can also be found in other class capsules. Besides, when we concatenate the capsules of these three classes together, three types of graphs can be well discriminated with the concatenated capsules which also directly reflect the classification performance. This property is quite different from standard scalar-based architectures where each graph is modeled with only one graph embedding 2. By introducing the concept of capsules, the graph and class capsules can not only preserve classification-related properties of each graph (reflected with the length of class capsules) but also other properties information (reflected with the angle of class capsules). The generated class capsules can also be useful in other follow-up work and we leave this to be explored in the future. We have proposed CapsGNN, a novel framework that fuses capsules theory into GNN for more efficient graph representation learning. Inspired by CapsNet, the concepts of capsules are introduced in this architecture to extract features in the form of vectors on the basis of nodes features extracted from GNN. As a , one graph is represented as multiple embeddings and each embedding captures different aspects of the graph properties. The generated graph and class capsules can preserve not only the classification-related information but also other information with respect to graph properties which might be useful in the follow-up work and we leave this to be explored in the future. We believe this is a novel, efficient and powerful data-driven method to represent high-dimensional data such as graphs. Our model has successfully achieved better or comparable performance when compared with other SOTA algorithms on 6 out of 10 graph classification tasks especially on social datasets. Compared with similar scalar-based architectures, CapsGNN is more efficient in encoding features and this would be very beneficial for processing large datasets. The specific procedure of routing is shown in Algorithm 1.Algorithm 1 Dynamic routing mechanism returns parent capsules H given children capsules S, a set of trainable transform matrices W and the number of iterations t.1: procedure DYNAMIC ROUTING(t, S, W)2:for all children capsule i: DISPLAYFORM0 for all children capsule i to all parent capsule j: r ij ← 0 4: DISPLAYFORM1 for all children capsule i:r i ← sof tmax(r i)6:for all parent capsule j: DISPLAYFORM2 for all parent capsule j: DISPLAYFORM3 for all children capsule i to all parent capsule j: r ij ← r ij +h DISPLAYFORM4 end for 10:returnh j 11: end procedure The details of benchmark datasets we use in our experiment is shown in TAB6. FIG3. This module is not necessary in some datasets. Here, we propose this module as a selective optimization. When the GNN goes deeper, the extracted nodes features contain more specific position information of each node. Inspired by where the node embeddings learned from the last layer of GNN are taken to order all nodes, we also take the capsules extracted from the last layer of GNN as the position indicators of corresponding nodes by concatenating it with each capsule of the node. The procedure of calculating votes with node position indicators can be written as: DISPLAYFORM0 where v (n,i)j ∈ R 1×(dn+dp) represents the node capsule vote from the ith channel of the nth node to the jth channel of graph capsules. W n ij ∈ R d×dn and W p j ∈ R d×dp are the transform matrices. s (n,i) is the same as introduced in Section 3.2 and represents concatenate operation. Here, we present a general comparison between CapsGNN with existing approaches. BID0, BID19 and (GNN-based graph representation learning architectures), CapsGNN represents node features in the form of capsules. This is helpful to preserve the properties information contained in nodes more efficiently when generating graph embeddings. Besides, each graph is modeled as multiple embeddings in CapsGNN instead of only one embedding used in other approaches. This allows us to capture information of graphs from different aspects. The second difference is that, in these approaches, each part of the graph is given equal importance. However, the attention mechanism used in CapsGNN allows it to assign various weights to different nodes. This leads the model to focus on critical parts of input graphs. Lastly, different from BID0 and BID19, uses node features extracted from multiple layers of GNN so that different size of receptive-fields are applied to preserve more information. 2) GCAPS-CNN proposed by BID22 also introduced capsule-related concept into graph representation learning. However, they generate capsules in a feature-based manner instead of learning capsules as distributed embeddings. More specifically, when they extend a scalar feature to a capsule for the node n, P higher-order statistical moment value is calculated based on its neighbor nodes and these P value is concatenated to a P -dimensional capsule. Between layers, GCAPS-CNN performs dimension reduction to compress capsules back to scalar features, which defeats the purpose of having capsules in the first place. CapsGNN learns each dimension of capsules in a data-driven manner and apply routing mechanism between layers to preserve the learned meaning of each capsule. This also allows us to preserve multiple properties information contained in nodes more efficiently especially when generating graph embeddings.3) Compared with CapsNet proposed by BID16 which works well in image processing domain, CapsGNN needs to handle more complex situations when handling graphs. In image processing domain, the size of the input images can be standardized by resizing the images. However, it is not possible to simply resize the graphs. So, we introduced an additional Attention Module to tackle graphs that are vastly different in sizes and preserve important parts of graphs. We also propose to use features extracted from all layers of GNN since it is hard to define a suitable receptive-field size for graphs. Furthermore, compared with the architecture of CapsNet, CapsGNN has one additional graph capsules layer which is used to learn multiple graph embeddings and these embeddings reflect different aspects of graph properties which is valuable in future research. To the best of our knowledge, we are the first one to model a graph as multiple embeddings in the form of distributed capsules and we believe this approach of learning representations has a high potential for other complex data analysis which is not limited to graphs. Besides, CapsGNN has different explanation of linear transformation. In CapsNet, by applying a linear trainable transformation to pose vectors, the spatial relationship between object parts and the whole object can be well modeled. However, by applying linear trainable transformation, CapsGNN is simply computing the prediction vectors from nodes-level representations to graph-level representations. This transform matrix is not trying to model the change of viewpoint or capture viewpoint invariant knowledge but to model the relationship between the properties of nodes and the properties of the whole graph. The same architecture settings are used in CapsGNN for all datasets to show its robust performance. For the node capsules extraction, the GCN has 5 layers (L = 5), the number of channels at each layer is set as the same which is 2 (C l = 2). The number of graph capsules is fixed as 16 (P = 16). The dimension of all capsules are set as 8 (d = 8). The number of units in the hidden layer of Attention Module is set as 1 16 of the number of input units. The number of iterations in routing is set as 3. During training stage, we simultaneously reduce Loss c and Loss r and we scale Loss r with 0.1 so that the model focuses on classification task. λ is set as 0.5 and 1.0 for multi-class classification and binary classification respectively. As for the node attributes in different datasets, considering that REDDIT-M5K and REDDIT-M12K are large-scale datasets with widely distributed nodes degree, we set the attributes of all nodes in these two datasets as the same which means we consider the initial node embeddings for all the nodes as the same to avoid over-fitting. For the remained relatively small datasets, both of the node degree and other node attributes are sent to CapsGNN as node features to speed up the training and we apply dropout(dropput rate is 0.3) to the input node features to improve the learning performance. The settings are summarized in the 3) CapsGNN-Avg (GCN + Average + Routing + Reconstruction): The Attention module in basic CapsGNN is replaced with the Average module.4) CapsGNN-noRout (GCN + Attention + Reconstruction): In this architecture, we will fix the similarity coefficients between all the capsules from one layer to the next layer as the same so that each children capsule will be equally routed to all the parent capsules.5) CapsGNN-noRecon (GCN + Attention + Routing): In this architecture, we directly remove the Reconstruction loss module.6) CapsGNN-Avg-noRout (GCN + Routing): In this architecture, we replace the Attention module in basic CapsGNN with the Average module and fix the similarity coefficients between all the capsules from one layer to the next layer as the same. The validation accuracy of each architecture is shown in TAB9 where we highlight the highest and lowest accuracy respectively. As we can see from the Table, IMDB-B and D&D reach better performance with CapsGNN-Coord which indicates the effectiveness of Coordinate Addition Module. However, the performance of social datasets is still comparable across all types of architectures. On the other hand, the performance of biological datasets(NCI1, PROTEINS, D&D) is more sensitive to the introduced modules in each architecture. The highest accuracy of NCI1, PROTEINS is achieved on CapsGNN which indicates the little effectiveness of Coordinate Addition Module in these two datasets. More specifically, the comparison between CapsGNN-Avg-noRout and CapsGNN-Avg on NCI1, PROTEINS and D&D indicates the effectiveness of Routing mechanism which improves the accuracy by 2.1%, 0.6% and 0.51% respectively. Besides, the comparison between CapsGNNAvg-noRout and CapsGNN-noRout on NCI1, PROTEINS and D&D indicates the effectiveness of Attention Module which improves the accuracy by 1.37%, 1.24% and 1.35% respectively. The accuracy of NCI1, PROTEINS and D&D can be improved by as much as 2.98%, 1.33% and 2.23% when Attention Module and Routing mechanism are combined in the architecture. Overall, CapsGNN is a general framework that fuses capsule theory to GNN for more efficient graph representation learning. In this framework, we also provide multiple possible modules to improve the quality of learned graph embeddings while we do not target to find the best combination of modules for each dataset. Since each possible module plays a different role in different datasets, it would be better to adjust the architecture and hyper-parameters based on practical situation.

represent the tldr for finding the corresponding scientific article

Inspired by CapsNet, we propose a novel architecture for graph embeddings on the basis of node features extracted from GNN.

Represent the TLDR for finding the corresponding scientific article:

We propose a new sparse and structured attention mechanism, TVmax, which promotes sparsity and encourages the weight of related adjacent locations to be the same.