Question:
represent the article for finding a single-sentence tldr
Fewshots:
'Capsule Networks have shown encouraging on \textit{defacto} benchmark computer vision datasets such as MNIST, CIFAR and smallNORB. Although, they are yet to be tested on tasks where the entities detected inherently have more complex internal representations and there are very few instances per class to learn from and where point-wise classification is not suitable. Hence, this paper carries out experiments on face verification in both controlled and uncontrolled settings that together address these points. In doing so we introduce \textit{Siamese Capsule Networks}, a new variant that can be used for pairwise learning tasks. We find that the model improves over baselines in the few-shot learning setting, suggesting that capsule networks are efficient at learning discriminative representations when given few samples. We find that \textit{Siamese Capsule Networks} perform well against strong baselines on both pairwise learning datasets when trained using a contrastive loss with $\ell_2$-normalized capsule encoded pose features, yielding best in the few-shot learning setting where image pairs in the test set contain unseen subjects. Convolutional Neural networks (CNNs) have been a mainstay model for a wide variety of tasks in computer vision. CNNs are effective at detecting local features in the receptive field, although the spatial relationship between features is lost when crude routing operations are performed to achieve translation invariance, as is the case with max and average pooling. Essentially, pooling in viewpoint invariance so that small perturbations in the input do not effect the output. This leads to a significant loss of information about the internal properties of present entities (e.g location, orientation, shape and pose) in an image and relationships between them. The issue is usually combated by having large amounts of annotated data from a wide variety of viewpoints, albeit redundant and less efficient in many cases. As noted by hinton1985shape, from a psychology perspective of human shape perception, pooling does not account for the coordinate frames imposed on objects when performing mental rotation to identify handedness BID23; BID19 BID12. Hence, the scalar output activities from local kernel regions that summarize sets of local inputs are not sufficient for preserving reference frames that are used in human perception, since viewpoint information is discarded. Spatial Transformer Networks (STN) BID13 have acknowledged the issue by using dynamic spatial transformations on feature mappings to enhance the geometric invariance of the model, although this approach addresses changes in viewpoint by learning to remove rotational and scale variance, as opposed to viewpoint variance being reflected in the model activations. Instead of addressing translation invariance using pooling operations, BID8 have worked on achieving translation equivariance. The recently proposed Capsule Networks BID24; BID7 have shown encouraging to address these challenges. Thus far, Capsule Networks have only been tested on datasets that have a relatively sufficient number of instances per class to learn from and utilized on tasks in the standard classification setup. This paper extends Capsule Networks to the pairwise learning setting to learn relationships between whole entity encodings, while also demonstrating their ability to learn from little data that can perform few-shot learning where instances from new classes arise during testing (i.e zero-shot prediction). The Siamese Capsule Network is trained using a contrastive loss with 2 -normalized encoded features and demonstrated on face verification tasks. BID8 first introduced the idea of using whole vectors to represent internal properties (referred to as instantiation parameters that include pose) of an entity with an associated activation probability where each capsule represents a single instance of an entity within in an image. This differs from the single scalar outputs in conventional neural networks where pooling is used as a crude routing operation over filters. Pooling performs sub-sampling so that neurons are invariant to viewpoint change, instead capsules look to preserve the information to achieve equivariance, akin to perceptual systems. Hence, pooling is replaced with a dynamic routing scheme to send lowerlevel capsule (e.g nose, mouth, ears etc.) outputs as input to parent capsule (e.g face) that represent part-whole relationships to achieve translation equivariance and untangles the coordinate frame of an entity through linear transformations. The idea has its roots in computer graphics where images are rendered given an internal hierarchical representation, for this reason the brain is hypothesized to solve an inverse graphics problem where given an image the cortex deconstructs it to its latent hierarchical properties. The original paper by BID24 describes a dynamic routing scheme that represent these internal representations as vectors given a group of designated neurons called capsules, which consist of a pose vector u ∈ R d and activation α ∈. The architecture consists of two convolutional layers that are used as the initial input representations for the first capsule layer that are then routed to a final class capsule layer. The initial convolutional layers allow learned knowledge from local feature representations to be reused and replicated in other parts of the receptive field. The capsule inputs are determined using a Iterative Dynamic Routing scheme. A transformation W ij is made to output vector u i of capsule C L i. The length of the vector u i represents the probability that this lower-level capsule detected a given object and the direction corresponds to the state of the object (e.g orientation, position or relationship to upper capsule). The output vector u i is transformed into a prediction vectorû j|i, whereû j|i = W ij u i. Then,û j|i is weighted by a coupling coefficient c ij to obtain s j = i c ijûj|i, where coupling coefficients for each capsule j c ij = 1 and c ij is got by log prior probabilities b ij from a sigmoid function, followed by the softmax, c ij = e bij / k e b ik. Ifû L j|i has high scalar magnitude when multiplied by u L+1 j then the coupling coefficient c ij is increased and the remaining potential parent capsules coupling coefficients are decreased. Routing By Agreement is then performed using coincidence filtering to find tight clusters of nearby predictions. The entities output vector length is represented as the probability of an entity being present by using the nonlinear normalization shown in Equation 1 where vote v j is the output from total input s j, which is then used to compute the agreement a ij = v jûj|i that is added to the log prior b ij. The capsule is assigned a high log-likelihood if densely connected clusters of predictions are found from a subset of s. The centroid of the dense cluster is output as the entities generalized pose. This coincidence filtering step can also be achieved by traditional outlier detection methods such as Random sample consensus (RANSAC) BID3 BID3 classical for finding subsets of the feature space with high agreement. Although, the motivation for using the vector normalization of the instantiation parameters is to force the network to preserve orientation. Lastly, a reconstruction loss on the images was used for regularization which constrains th capsules to learn properties that can better encode the entities. In this paper, we do not use such regularization scheme by autoencoding pairs of input images, instead we use a variant of dropout. BID7 recently describe matrix capsules that perform routing by agreement using the expectation maximization (EM) algorithm, motivated by computer graphics where pose matrices are used to define rotations and translations of objects to account for viewpoint changes. Each parent capsule is considered a Gaussian and the pose matrix of each child capsule are considered data samples of the Gaussian. A given layer L contains a set of capsules C L such that ∀C ∃ {M, α} ∈ C L where pose matrix M ∈ R n×n (n = 4) and activation α ∈ are the outputs. A vote is made V ij = M i W ij for the pose matrix of C L+1 j where W ij ∈ R n×n is a learned viewpoint invariant transformation matrix from capsule where the cost h j is the negative log-probability density weighted by the assignment probabilities r ij, −β u is the negative log probability density per pose matrix computed to describe C L+1 j. If C L+1 j is activated −β a is the cost for describing (µ j, σ 2 j) from lower-level pose data samples along with r ij and λ is the inverse temperature so as the assignment probability becomes higher the slope of the sigmoid curve becomes steeper (represents the presence of an entity instead of the nonlinear vector normalization seen in Equation 1). The network uses 1 standard convolutional layer, a primary capsule layer, 2 intermediate capsule convolutional layer, followed by the final class capsule layer. The matrix capsule network significantly outperformed CNNs on the SmallNORB dataset. LaLonde & Bagci FORMULA7 introduce SegCaps which uses a locally connected dynamic routing scheme to reduce the number of parameters while using deconvolutional capsules to compensate for the loss of global information, showing best performance for segmenting pathological lungs from low dose CT scans. The model obtained a 39% and 95% reduction in parameters over baseline architectures while outperforming both. Bahadori FORMULA7 introduced Spectral Capsule Networks demonstrated on medical diagnosis. The method shows faster convergence over the EM algorithm used with pose vectors. Spatial coincidence filters align extracted features on a 1-d linear subspace. The architecture consists of a 1d convolution followed by 3 residual layers with dilation. Residual blocks R are used as nonlinear transformations for the pose and activation of the first primary capsule instead of the linear transformation that accounts for rotations in CV, since deformations made in healthcare imaging are not fully understood. The weighted votes are obtained as s j,i = α i R j (u i) ∀i where S j is a matrix of concatenated votes that are then decomposed using SVD, where the first singular value dimensions 1 is used to capture most of the variance between votes, thus the activation a j activation is computed as σ η(s DISPLAYFORM0 k is the ratio of all variance explained for all right singular vectors in V, b is optimized and η is decreased during training. The model is trained by maximizing the log-likelihood showing better performance than the spread loss used with matrix capsules and mitigates the problem of capsules becoming dormant. BID31 formalize the capsule routing strategy as an optimization of a clustering loss and a KL regularization term between the coupling coefficient distribution and its past states. The proposed objective function follows as min C,S {L(C, S):= − i j c ij o j|i, s j + α i j c ij log c ij } where o j|i = T ij µ i /||T ij || F and ||T ij || F is the Frobenious norm of T ij. This routing scheme shows significant benefit over the original routing scheme by BID24 as the number of routing iterations increase. Evidently, there has been a surge of interest within the research community. In contrast, the novelty presented in this paper is the pairwise learning capsule network scheme that proposes a different loss function, a change in architecture that compares images, aligns entities across images and describes a method for measuring similarity between final layer capsules such that inter-class variations are maximized and intra-class variations are minimized. Before describing these points in detail, we briefly describe the current state of the art work (SoTA) in face verification that have utilized Siamese Networks. Siamese Networks (SNs) are neural networks that learn relationships between encoded representations of instance pairs that lie on low dimensional manifold, where a chosen distance function d ω is used to find the similarity in output space. Below we briefly describe state of the art convolutional SN's that have been used for face verification and face recognition. BID27 presented a joint identification-verification approach for learning face verification with a contrastive loss and face recognition using cross-entropy loss. To balance loss signals for both identification and verification, they investigate the effects of varying weights controlled by λ on the intra-personal and inter-personal variations, where λ = 0 leaves only the face recognition loss and λ → ∞ leaves the face verification loss. Optimal are found when λ = 0.05 intra personal variation is maximized while both class are distinguished. BID32 propose a center loss function to improve discriminative feature learning in face recognition. The center loss function proposed aims to improve the discriminability between feature representations by minimizing the intra-class variation while keeping features from different classes separable. The center loss is given as DISPLAYFORM0 where z = W T j x i + b j. The c yi is the centroid of feature representations pertaining to the i th class. This penalizes the distance between class centers and minimizes the intra-class variation while the softmax keeps the inter-class features separable. The centroids are computed during stochastic gradient descent as full batch updates would not be feasible for large networks. BID18 proposed Sphereface, a hypersphere embedding that uses an angular softmax loss that constrains disrimination on a hypersphere manifold, motivated by the prior that faces lie on a manifold. The model achieves 99.22 % on the LFW dataset, and competitive on Youtube Face (YTF) and MegaFace. BID25 proposed a triplet similarity embedding for face verification using a triple loss arg min W = α,p,n∈T max(0, α + α T W T W (n − p)) where for T triplet sets lies an anchor class α, positive class p and negative class n, a projection matrix W, (performed PCA to obtain W 0) is minimized with the constraint that W BID9 use deep metric learning for face verification with loss arg min DISPLAYFORM1 DISPLAYFORM2 F | where g(z) = log(1 + e βz)/β, β controls the slope steepness of the logistic function, ||A|| F is the frobenius norm of A and λ is a regularization parameter. Hence, the loss function is made up of a logistic loss and regularization on parameters θ = [W, b]. Best are obtained using a combination of SIFT descriptors, dense SIFT and local binary patterns (LBP), obtaining 90.68% (+/-1.41) accuracy on the LFW dataset. BID21 used an 2 -constraint on the softmax loss for face verification so that the encoded face features lie on the ambit of a hypersphere, showing good improvements in performance. This work too uses an 2 -constraint on capsule encoded face embeddings. FaceNet BID26 too uses a triplet network that combines the Inception network BID28 and a 8-layer convolutional model BID33 which learns to align face patches during training to perform face verification, recognition and clustering. The method trains the network on triplets of increasing difficulty using a negative example mining technique. Similarly, we consider a Siamese Inception Network for the tasks as one of a few comparisons to SCNs. The most relevant and notable use of Siamese Networks for face verification is the DeepFace network BID30. The performance obtained was on par with human level performance on the Faces in the Wild (LFW) dataset and significantly outperformed previous methods. However, it is worth noting this model is trained on a large dataset from Facebook (SFC), therefore the model can be considered to be performing transfer learning before evaluation. The model also carries out some manual steps for detecting, aligning and cropping faces from the images. For detecting and aligning the face a 3D model is used. The images are normalized to avoid any differences in illumination values, before creating a 3D model which is created by first identifying 6 fiducial points in the image using a Support Vector Regressor from a LBP histogram image descriptor. Once the faces are cropped based on these points, a further 67 fiducial point are identified for 3D mesh model, followed by a piecewise affine transformation for each section of the image. The cropped image is then passed to 3 CNN layers with an initial max-pooling layer followed two fully-connected layers. Similar to Capsule Networks, the authors refrain from using max pooling at each layer due to information loss. In contrast to this work, the only preprocessing steps for the proposed SCNs consist of pixel normalization and a reszing of the image. The above work all achieve comparable state of the art for face verification using either a single CNN or a combination of various CNNs, some of which are pretrained on large related datasets. In contrast, this work looks to use a smaller Capsule Network that is more efficient, requires little preprocessing steps (i.e only a resizing of the image and normalization of input features, no aligning, cropping etc.) and can learn from relatively less data. The Capsule Network for face verification is intended to identify enocded part-whole relationships of facial features and their pose that in turn leads to an improved similarity measure by aligning capsule features across paired images. The architecture consists of a 5-hidden layer (includes 2 capsule layers) network with tied weights (since both inputs are from the same domain). The 1 st layer is a convolutional filter with a stride of 3 and 256 channels with kernels κ 1 i ∈ R 9×9 ∀i over the image pairs x 1, x 2 ∈ R 100×100, ing in 20, 992 parameters. The 2 nd layer is the primary capsule layer that takes κ and outputs κ ∈ R 31×31 matrix for 32 capsules, leading to 5.309 × 10 6 parameters (663, 552 weights and 32 biases for each of 8 capsules). The 3 rd layer is the face capsule layer, representing the routing of various properties of facial features, consisting of 5.90 × 10 6 parameters. This layer is then passed to a single fully connected layer by concatenating DISPLAYFORM0 as input, while the sigmoid functions control the dropout rate for each capsule during training. The nonlinear vector normalization shown in Equation 1 is replaced with a tanh function tanh which we found in initial testing to produce better . Euclidean distance, Manhattan distance and cosine similarity are considered as measures between the capsule image encodings. The aforementioned SCN architecture describes the setup for the AT&T dataset. For the LFW dataset, 6 routing iterations are used and 4 for AT&T. To encode paired images x 1, x 2 into vector pairs h 1, h 2 the pose vector of each capsule is vectorized and passed as input to a fully connected layer containing 20 activation units. Hence, for each input there is a lower 20-dimensional representation of 32 capsule pose vectors ing in 512 input features. To ensure all capsules stay active the dropout probability rate is learned for each capsule. The logit function learns the dropout rate of the final capsule layer using Concrete Dropout BID4. This builds on prior work Kingma et al. FORMULA7; BID20 by using a continuous relaxation that is referred to as a concrete distribution, the mask becomes differentiable, unlike the discrete Bernoulli distribution used in Binary dropout. Equation 2 shows the objective function for updating the concrete distribution. For a given capsule probability p c in the last capsule layer, the sigmoid computes the relaxationz on the Bernoulli variable z, where u is drawn uniformly between and log(u c) produces log-uniform noise, and t denotes the temperature values (t = 0.1 in our experiments) which forces probabilities at the extremum when small. The pathwise derivative estimator is used to find a continuous estimation of the dropout mask.z DISPLAYFORM0 Equation FORMULA8 show the dropout loss where the LHS numerator represents the weight regularization term for N weights in the final layer L where γ controls the amount of regularization. The remaining H(p d L) is the dropout entropy term used as dropout regularization. This entropy term H(p d L) is essential to learn the dropout probability for capsule representations, also note when regularization plays a significant role reducing the contrastive loss, d p → 0.5. In our experiments, weight regularization λ ω L = 0.1 is chosen based on a grid search during training for λ θ L = {0.001, 0.01, 0.1, 0.2, 0.5}. This is used as an auxiliary regularization term with the contrastive losses discussed in the next section. DISPLAYFORM1 We also use a straight-through pathwise derivative estimator BID5 to then optimize the dropout probability, following the implementation by BID4. BID7 has also been used to maximize the inter-class distance between the target class and the remaining classes for classifying on the smallNORB dataset. This is given as where the margin m is increased linearly during training to ensure lower-level capsule stay active throughout training. This work instead uses a contrastive margin loss BID2 where the aforementioned capsule encoding similarity function d ω outputs a predicted similarity score. The contrastive loss L c ensures similar vectorized pose encodings are drawn together and dissimilar poses repulse. Equation 4 shows a a pair of images that are passed to the SCN model where DISPLAYFORM0 2 computes the Euclidean distance between encodings and m is the margin. When using Manhattan distance DISPLAYFORM1 A double margin loss that has been used in prior work by BID17 is also considered to affect matching pairs such that to account for positive pairs that can also have high variance in the distance measure. It is worth noting this double margin is similar to the aforementioned margin loss used on class capsules, without the use of λ. Equation 5 shows the double-margin contrastive loss where positive margin m p and negative margin m n are used to find better separation between matching and non-matching pairs. This loss is only used for LFW, given the limited number of instances in AT&T we find the amount of overlap between pairs to be less severe in experimentation. DISPLAYFORM2 The original reconstruction loss DISPLAYFORM3 i ) 2 used as regularization is not used in the pairwise learning setting, instead we rely on the dropout for regularization with exception of the SCN model that uses concrete dropout on the final layer. In the case of using concrete dropout the loss is L conc = L c (ω) + Γ c (p d) and Γ controls the amount of dropout regularization Optimization Convergence can often be relatively slow for face verification tasks, where few informative batch updates (e.g a sample with significantly different pose for a given class) get large updates but soon after the effect is diminished through gradient exponential averaging (originally introduced to prevent α → 0). Motivated by recent findings that improve adaptive learning rates we use AMSGrad BID22. AMSGrad improves over ADAM in some cases by replacing the exponential average of squared gradients with a maximum that mitigates the issue by keeping long-term memory of past gradients. Thus, AMSGrad does not increase or decrease the learning rate based on gradient changes, avoiding divergent or vanishing step sizes over time. Equation 6 presents the update rule, where diagonal of gradient g t is given as DISPLAYFORM4 A. AT&T dataset The AT&T face recognition and verification dataset consists of 40 different subjects with only 10 gray-pixel images per subject in a controlled setting. This smaller dataset allows us to test how SCNs perform with little data. For testing, we hold out 5 subjects so that we are testing on unseen subjects, as opposed to training on a given viewpoint of a subject and testing on another viewpoint of the same subject. Hence, zero-shot pairwise prediction is performed during testing. The LFW consists of 13,000 colored photographed faces from the web. This dataset is significantly more complex not only because there 1680 subjects, with some subjects only consisting of two images, but also because of varied amount of aging, pose, gender, lighting and other such natural characteristics. Each image is 250 × 250, in this work the image is resized to 100×100 and normalized. From the original LFW dataset there has been 2 different versions of the dataset that align the images using funneling BID10 and deep funneling BID11. The latter learns to align the images using Restricted Boltzmann Machines with a group sparsity penalty, showing performance improvements for face verification tasks. The penalty leads to an arrangement of the filters that improved the alignment . This overcomes the problems previous CNNs and models alike had in accounting for pose, orientation and problems Capsule Networks look to address. In contrast, we use the original raw image dataset. Both datasets allow us to test different challenges in face verification. The AT&T dataset provides a useful benchmark for the siamese-based models in the few-shot learning setting as it only provides a limited number of instances and unique subjects, while also being limited to grey-pixel images. In contrast, LFW contains coloured images that are in an unconstrained setting, large in comparison to AT&T and have highly imbalanced classes. Baselines SCNs are compared against well-established architectures for image recognition and verification tasks, namely AlexNet BID15, ResNet-34 and InceptionV3 BID29 with 6 inception layers instead of the original network that uses 8 layers which are used many of the aforementioned papers in Section 3. Table 1 shows best test obtained when using contrastive loss with Euclidean distance between encodings (i.e Mahalanobis distance) for both AT &T and LFW over 100 epochs. Shaded cells correspond to best performing models over test contrastive loss and test accuracy for each respective dataset and loss (e.g LFW+Double Margin (M)). The former uses m = 0.4 and the latter uses m = 0.2, while for the double margin contrastive loss m n = 0.2 matching margin and m p = 0.5 negative matching margin is selected. These settings were chosen during 5-fold cross validation, grid searching over possible margin settings. SCN outperforms baselines on the AT &T dataset after training for 100 epochs. We find that because AT&T contains far fewer instances an adapted dropout rate leads to a slight increase in contrastive loss. Additionally, adding a reconstruction loss with λ r = 1e −4 for both paired images led to a decrease in performance when compared to using dropout with a rate p = 0.2 on all layers except the final layer that encodes the pose vectors. We find for the LFW dataset that the SCN and AlexNet have obtained the best while SCN has 25% less parameters. Additionally, the use of a double margin in better for the LeNet SCN but a slight drop in performance when used with concrete dropout on the final layer (i.e SDropCapNet). Figure 1 illustrates the contrastive loss during training 2 -normalized features for each model tested with various distance measures on AT&T and LFW. We find that SCN yields faster convergence on AT&T, particularly when using Manhattan distance. However for Euclidean distance, we observe a loss variance reduction during training and the best overall performance. Through experiments we find that batch normalized convolutional layers improves performance of the SCN. In batch normalization, (k) and scaled with β (k) so that a (k) = γ (k)x(k) + β (k). This allows the network to learn whether the input range should be more or less diffuse. Batch normalization on the initial convolutional layers reduced variance in loss during training on both the AT &T and LF W datasets. LFW test show that the SCN model takes longer to converge particularly in the early stages of training, in comparison to AlexNet. Figure 2 shows the probability density of the positive pair predictions for each model for all distances between encodings with contrastive loss for the LFW dataset. Accuracy is computed using the aforementioned margins to separate positive and negatives instance pairs, which is set to be constant throughout training since the capsule encoded features are − 2 normalized, similar to the approach taken for FaceNet BID26. We find the variance of predictions is lower in comparison to the remaining models, showing a higher precision in the predictions, particularly for Manhattan distance. Additionally, varying distances for these matching images were close in variance to nonmatching images. This motivated the use of the double margin loss considered for the LFW dataset. DISPLAYFORM0 Finally, the SCN model has between 104-116 % less parameters than Alexnet, 24-27 % Resnet-34 and 127-135% less than the best LeNet baseline for both datasets. However, even considering tied weights between models in the SCN, Capsule Networks are primarily limited in speed even with a reduction in parameters due to the routing iterations that are necessary during training. This paper has introduced the Siamese Capsule Network, a novel architecture that extends Capsule Networks to the pairwise learning setting with a feature 2 -normalized contrastive loss that maximizes inter-class variance and minimizes intra-class variance. The indicate Capsule Networks perform better at learning from only few examples and converge faster when a contrastive loss is used that takes face embeddings in the form of encoded capsule pose vectors. We find Siamese Capsule Networks to perform particularly well on the AT&T dataset in the few-shot learning setting, which is tested on unseen classes (i.e subjects) during testing, while competitive against baselines for the larger Labeled Faces In The Wild dataset.' == 'A pairwise learned capsule network that performs well on face verification tasks given limited labeled data ' != 'We generate examples to explain a classifier desicion via interpolations in latent space. The variational auto encoder cost is extended with a functional of the classifier over the generated example path in data space.'

Question: We propose Support-guided Adversarial Imitation Learning (SAIL), a generic imitation learning framework that unifies support estimation of the expert policy with the family of Adversarial Imitation Learning (AIL) algorithms. SAIL addresses two important challenges of AIL, including the implicit reward bias and potential training instability. We also show that SAIL is at least as efficient as standard AIL. In an extensive evaluation, we demonstrate that the proposed method effectively handles the reward bias and achieves better performance and training stability than other baseline methods on a wide range of benchmark control tasks. The class of Adversarial Imitation Learning (AIL) algorithms learns robust policies that imitate an expert's actions from a small number of expert trajectories, without further access to the expert or environment signals. AIL iterates between refining a reward via adversarial training, and reinforcement learning (RL) with the learned adversarial reward. For instance, Generative Adversarial Imitation Learning (GAIL) shows the equivalence between some settings of inverse reinforcement learning and Generative Adversarial Networks (GANs) , and recasts imitation learning as distribution matching between the expert and the RL agent. Similarly, Adversarial Inverse Reinforcement Learning (AIRL) modifies the GAIL discriminator to learn a reward function robust to changes in dynamics or environment properties. AIL mitigates the issue of distributional drift from behavioral cloning , a classical imitation learning algorithm, and demonstrates good performance with only a small number of expert demonstrations. However, AIL has several important challenges, including implicit reward bias , potential training instability , and potential sample inefficiency with respect to environment interaction . In this paper, we propose a principled approach towards addressing these issues. demonstrated that imitation learning is also feasible by constructing a fixed reward function via estimating the support of the expert policy. Since support estimation only requires expert demonstrations, the method sidesteps the training instability associated with adversarial training. However, we show in Section 4.2 that the reward learned via support estimation deteriorates when expert data is sparse, and leads to poor policy performances. Support estimation and adversarial reward represent two different yet complementary RL signals for imitation learning, both learnable from expert demonstrations. We unify both signals into Supportguided Adversarial Imitation Learning (SAIL), a generic imitation learning framework. SAIL leverages the adversarial reward to guide policy exploration and constrains the policy search to the estimated support of the expert policy. It is compatible with existing AIL algorithms, such as GAIL and AIRL. We also show that SAIL is at least as efficient as standard AIL. In an extensive evaluation, we demonstrate that SAIL mitigates the implicit reward bias and achieves better performance and training stability against baseline methods over a series of benchmark control tasks. We briefly review the Markov Decision Process (MDP), the context of our imitation learning task, followed by related works on imitation learning. We consider an infinite-horizon discounted MDP (S, A, P, r, p 0, γ), where S is the set of states, A the set of actions, P: S × A × S → the transition probability, r: S × A → R the reward function, p 0: S → the distribution over initial states, and γ ∈ the discount factor. Let π be a stochastic policy π:, and s t+1 ∼ P (·|s t, a t) for t ≥ 0. We denote π E the expert policy. Behavioral Cloning (BC) learns a policy π: S → A directly from expert trajectories via supervised learning. BC is simple to implement, and effective when expert data is abundant. However, BC is prone to distributional drift: the state distribution of expert demonstrations deviates from that of the agent policy, due to accumulation of small mistakes during policy execution. Distributional drift may lead to catastrophic errors . While several methods address the issue , they often assume further access to the expert during training. Inverse Reinforcement Learning (IRL) first estimates a reward from expert demonstrations, followed by RL using the estimated reward . Building upon a maximum entropy formulation of IRL , and explore adversarial IRL and its connection to Generative Adversarial Imitation Learning . Imitation Learning via Distribution Matching Generative Adversarial Imitation Learning (GAIL) frames imitation learning as distribution matching between the expert and the RL agent. The authors show the connection between IRL and GANs. Specifically, GAIL imitates the expert by formulating a minimax game: min where the expectations E π and E π E denote the joint distributions over state-actions of the RL agent and the expert, respectively. GAIL is able to achieve expert performance with a small number of expert trajectories on various benchmark tasks. However, GAIL is relatively sample inefficient with respect to environment interaction, and inherits issues associated with adversarial learning, such as vanishing gradients, training instability and overfitting to expert demonstrations . Recent works have improved the sample efficiency and stability of GAIL. For instance, Generative Moment Matching Imitation Learning replaces the adversarial reward with a non-parametric maximum mean discrepancy estimator to sidestep adversarial learning. improve sample efficiency with a model-based RL algorithm. and demonstrate significant gain in sample efficiency with off-policy RL algorithms. In addition, Generative Predecessor Models for Imitation Learning imitates the expert policy using generative models to reason about alternative histories of demonstrated states. Our proposed method is closely related to the broad family of AIL algorithms including GAIL and adversarial IRL. It is also complementary to many techniques for improving the algorithmic efficiency and stability, as discussed above. In particular, we focus on improving the quality of the learned reward by constraining adversarial reward to the estimated support of the expert policy. Imitation Learning via Support Estimation Alternative to demonstrate the feasibility of using a fixed RL reward via estimating the support of the expert policy from expert demonstrations. Connecting kernel-based support estimation to Random Network Distillation , the authors propose Random Expert Distillation (RED) to learn a reward function based on support estimation. Specifically, RED learns the reward parameterθ by minimizing: where f θ: S × A → R K projects (s, a) from expert demonstrations to some embedding of size K, with randomly initialized θ. The reward is then defined as: where σ is a hyperparameter. As optimizing Eq. only requires expert data, RED sidesteps adversarial learning, and casts imitation learning as a standard RL task using the learned reward. While RED works well given sufficient expert data, we show in the experiments that its performance suffers in the more challenging setting of sparse expert data. Formally, we consider the task of learning a reward functionr(s, a) from a finite set of trajectories, sampled from the expert policy π E within a MDP. Each trajectory is a sequence of stateaction tuples in the form of τ i = {s 1, a 1, s 2, a 2, ..., s T, a T}. Assuming that the expert trajectories are consistent with some latent reward function r * (s, a), we aim to learn a policy that achieves good performance with respect to r * (s, a) by applying RL on the learned reward functionr(s, a). In this section, we first discuss the advantages and shortcomings of AIL to motivate our method. We then introduce Support-guided Adversarial Learning (SAIL), and present a theoretical analysis that compares SAIL with the existing methods, specifically GAIL. A clear advantage of AIL resides in its low sample complexity with respect to expert data. For instance, GAIL requires as little as 200 state-action tuples from the expert to achieve imitation. The reason is that the adversarial reward may be interpreted as an effective exploration mechanism for the RL agent. To see this, consider the learned reward function under the optimality assumption. With the optimal discriminator to Eq. Eq. shows that the adversarial reward only depends on the ratio φ(s, a) = pπ(s,a). Intuitively, r gail incentivizes the RL agent towards under-visited state-actions, where φ(s, a) > 1, and away from over-visited state-actions, where φ(s, a) < 1. When π E and π match exactly, r gail converges to an indicator function for the support of π E, since φ(s, a) = 1 ∀ (s, a) ∈ supp(π E) . In practice, the adversarial reward is unlikely to converge, as p π E is estimated from a finite set of expert demonstrations. Instead, the adversarial reward continuously drives the agent to explore by evolving the reward landscape. However, AIL also presents several challenges. demonstrated that the reward − log D(s, a) suffers from an implicit survival bias, as the non-negative reward may lead to suboptimal behaviors in goal-oriented tasks where the agent learns to move around the goal to accumulate rewards, instead of completing the tasks. While the authors resolve the issue by introducing absorbing states, the solution assumes extra RL signals from the environment, including access to the time limit of an environment to detect early termination of training episodes. In Section 4.1, we empirically demonstrate the survival bias on Lunar Lander, a common RL benchmark, by showing that agents trained with GAIL often hover over the goal location 1. We also show that our proposed method is able to robustly imitate the expert. Another challenge with AIL is potential training instability. demonstrated empirically that the adversarial reward could be unreliable in regions where the expert data is sparse, causing the agent to diverge from the intended behavior. When the agent policy is substantially different from the expert policy, the discriminator could differentiate them with high confidence, ing in very low rewards and significant slow down in training, similar to the vanishing gradient problem in GAN training . We propose a novel reward function by combining the standard adversarial reward r gail with the corresponding support guidance r red. SAIL is designed to leverage the exploration mechanism offered by the adversarial reward, and to constrain the agent to the estimated support of the expert policy. Despite being a simple modification, support guidance provides strong reward shaping to address the challenges discussed in the previous, Θ function models, initial policy π ω0, initial discriminator parameters w 0, learning rate l D. 2: r red = RED(Θ, τ E) 3: for i = 0, 1,... sample a trajectory τ i ∼ π 5: π ωi+1 = TRPO(r red · r gail, π ωi). Sample θ ∈ Θ 10:θ =MINIMIZE(fθ, f θ, τ) 11: section. As both support guidance and adversarial reward are learnable from expert demonstrations, our method requires no further assumptions that standard AIL. SAIL addresses the survival bias in goal-oriented tasks by encouraging the agent to stop at the goal and complete the task. In particular, r red shapes the adversarial reward by favoring stopping at the goal against all other actions, as stopping at the goal is on the support of the expert policy, while other actions are not. We demonstrate empirically that SAIL assigns significantly higher reward towards completing the task and corrects for the bias in Section 4.1. To improve training stability, SAIL constrains the RL agent to the estimated support of the expert policy, where r gail provides a more reliable RL signal . As r red tends to be very small (ideally zero) for (s, a) ∈ supp(π E), r sail discourages the agent from exploring those state-actions by masking away the rewards. This is a desirable property as the quality of the RL signals beyond the support of the expert policy can't be guaranteed. We demonstrate in Section 4.2 the improved training stability on the Mujoco benchmark tasks. We provide the pseudocode implementation of SAIL in Algorithm 1. The algorithm computes r red by estimating the support of the expert policy, followed by iterative updates of the policy and r gail. We apply the Trust Region Policy Optimization (TRPO) algorithm with the reward r sail for policy updates. Reward Variants In practice, we observe that constraining the range of the adversarial reward generally produces lower-variance policies. Specifically, we transform r gail in Eq. For ease of notation, we refer to the bounded variant as SAIL-b, and the unbounded variant as SAIL. Similarly, we denote the bounded GAIL reward as GAIL-b. We include the comparison between the reward variants in the experiments. In this section, we show that SAIL is at least as efficient as GAIL in its sample complexity for expert data, and provide comparable RL signals on the expert policy's support. We note that our analysis could be similarly applied to other AIL methods, suggesting the broad applicability of our approach. We begin from the asymptotic setting, where the number of expert trajectories tends to infinity. In this case, both GAIL's, RED's and SAIL's discriminators ultimately recover the expert policy's support at convergence (see for for RED; SAIL follows from their combination). Moreover, for both GAIL and SAIL, the expert and agent policy distributions match exactly at convergence, implying a successful imitation learning. Therefore, it is critical to characterize the rates of convergence of the two methods, namely their relative sample complexity with respect to the number of expert demonstrations. Formally, let (s, a) ∈ supp(π E). Prototypical learning bounds for an estimator of the support r ≥ 0 provide high probability bounds in the form of P(r(s, a) ≤ c log(1/δ)n −α ) > 1 − δ for any confidence δ ∈, with c a constant not depending on δ or the number n of samples (i.e., expert state-actions). Here, α > 0 represents the learning rate, namely how fast the estimator is converging to the support. By choosing the reward in Eq., we are leveraging the faster learning rates between α red and α gail, with respect to support estimation. At the time being, no are available to characterize the sample complexity of GAIL (loosely speaking, the α and c introduced above). Therefore, we proceed by focusing on a relative comparison with SAIL. In particular, we show the following (see appendix for a proof). Proposition 1. Assume that for any (s, a) ∈ supp(π E) the rewards for RED and GAIL have the following learning rates in estimating the support Then, for any δ ∈ and any (s, a) ∈ supp(π E), the following holds with probability at least 1 − δ, where R red and R gail are the upper bounds for r red and r gail, respectively. Eq. shows that SAIL is at least as fast as the faster among RED and GAIL with respect to support estimation, implying that SAIL is at least as efficient as GAIL in the sample complexity for expert data. Eq. also indicates the quality of the learned reward, as state-actions outside the expert's support should be assigned minimum reward. Proposition 2. For any (s, a) ∈ supp(π E) and any δ ∈, we assume that The following event holds with probability at least 1 − δ that Eq. shows that on the expert policy's support, r sail is close to r gail up to a precision that improves with the number of expert state-actions. SAIL thus provides RL signals comparable to GAIL on the expert policy's support. It is also worth noting that the analysis could explain why r red + r gail is a less viable approach for combining the two RL signals. The analogous bound to Eq. would be the sum of errors from the two methods, implying the slower of the two learning rates, while Eq. would improve only by a constant, as R gail would be absent from Eq.. Our preliminary experiments indicated that r red + r gail performed noticeably worse than Eq.. Lastly, we comment on whether the assumptions in Eqs. and are satisfied in practice. Following the kernel-based version of RED , we can borrow previous from the set learning literature, which guarantee RED to have a rate of α red = 1/2 . These rates have been shown to be optimal. Any estimator of the support cannot have faster rates than n −1/2, unless additional assumptions are imposed. Learning rates for distribution matching with GANs are still an active area of research, and conclusive characterizing the convergence rates of these estimators are not available. We refer to for an in-depth analysis of the topic. We evaluate the proposed method against BC, GAIL and RED on Lunar Lander and six Mujoco control tasks including Hopper, Reacher, HalfCheetah, Walker2d, Ant, and Humanoid. We omit evaluation against methods using off-policy RL algorithms, as they are not the focus of this work. We also note that support guidance is complementary to such methods. We demonstrate that SAIL variants mitigate the survival bias in Lunar Lander (Fig. 1) from OpenAI Gym , while other baseline methods imitate the expert inconsistently. In this task, the agent is required to control a spacecraft to safely land between the flags. A human expert provided 10 demonstrations for this task as an imitation target. We observe that even without the environment reward, Lunar Lander provides a natural RL signal by terminating episodes early when crashes are detected, thus encouraging the agent to avoid crashing. Consequently, all methods are able to successfully imitate the expert and land the spacecraft appropriately. SAIL variants perform slightly better than GAIL variants on the average reward, and achieve noticeably lower standard deviation. The average performances and the standard deviations evaluated over 50 runs are presented in Table 1. To construct a more challenging task, we disable all early termination feature of the environment, thus removing the environment RL signals. In this no-terminal environment, a training episode only ends after the time limit. We present each algorithm's performance for the no-terminal setting in Table 1. SAIL variants outperform GAIL variants. Specifically, we observe that GAIL learns to land for some initial conditions, while exhibit survival bias in other scenarios by hovering at the goal. In contrast, SAIL variants are still able to recover the expert policy. To visualize the shaping effect from support guidance, we plot the average learned reward for GAIL, SAIL-b and RED at goal states. The goal states are selected from the expert trajectories and satisfy two conditions: 1) touching the ground (the state vector has indicator variables for ground contact), and 2) has "no op" as the corresponding action. As the adversarial reward functions are dynamic, we snapshot the learned rewards when the algorithms obtain their best policies, respectively. Fig. 3 shows the average rewards for each available action, averaged across all the goal states. Compared against the other algorithms, SAIL-b assigns a significantly higher reward to "no op", which facilitates the agent learning. Though GAIL and RED still favor "no op" to other actions, the differences in reward are much smaller, causing less consistent landing behaviors. We further observe that all evaluated AIL methods oscillate between partially hovering behavior and landing behavior during policy learning. The observation suggests that our method only partially addresses the survival bias, a limitation we will tackle in future works. This is likely caused by SAIL's non-negative reward, despite the beneficial shaping effect from support estimation. For additional experiment and discussion on Lunar Lander, please refer to the appendix. Mujoco control tasks have been commonly used as the standard benchmark for AIL. We evaluate SAIL against GAIL, RED and BC on Hopper, Reacher, HalfCheetah, Walker2d, Ant and Humanoid. We adopt the same experimental setup presented in by sub-sampling the expert trajectories every 20 samples. Consistent with the observation from , our preliminary experiments show that sub-sampling presents a more challenging setting, as BC is competitive with AIL when full trajectories are used. In our experiments, we also adopt the minimum 1056.5 ± 0.5 -9.1 ± 4.1 -0.2 ± 0.7 2372.8 ± 8.8 1005.5 ± 8.6 6012.0 ± 434.9 GAIL 3826.5 ± 3.2 -9.1 ± 4.4 4604.7 ± 77.6 5295.4 ± 44.1 1013.3 ± 16.0 8781.2 ± 3112.6 GAIL-b 3810.5 ± 8.1 -8.3 ± 2.5 4510.0 ± 68.0 5388.1 ± 161.2 3413.1 ± 744.7 10132.5 ± 1859.3 SAIL 3824.7 ± 6.6 -7.5 ± 2.7 4747.5 ± 43.4 5293.0 ± 590.9 3330.4 ± 729.4 9292.8 ± 3190.0 SAIL-b 3811.6 ± 3.8 -7.4 ± 2.5 4632.2 ± 59.1 5438.6 ± 18.4 4176.3 ± 203.1 10589.6 ± 52.2 Table 2: Episodic reward and standard deviation on the Mujoco tasks by different methods evaluated over 50 runs. SAIL-b achieves overall the best performance, with significantly lower standard deviation, indicating the robustness of the learned policies. number of expert trajectories specified in for each task. More details on experiment setup are available in the appendix. We apply each algorithm using 5 different random seeds in all Mujoco tasks. Table 2 shows the performance comparison between the evaluated algorithms. We report the mean performance and standard deviation for each algorithm over 50 evaluation runs, choosing the best policies obtained for each algorithm out of the 5 random seeds. The show that SAIL-b is comparable to GAIL on Hopper, and outperform the other methods on all other tasks. We note that RED significantly underperforms in the sub-sampling setting, while used full trajectories in their experiments. Across all tasks, SAIL-b generally achieves lower standard deviation compared to other algorithms, in particular for Humanoid, indicating the robustness of the learned policies. We stress that standard deviation is also a critical metric, as it indicates the robustness of the learned policies when presented with different states. For instance, the large standard deviations in Humanoid are caused by occasional crashes, which may be highly undesirable depending on the intended applications. To illustrate robustness of the learned policies, we plot the histogram of all 50 evaluations in Humanoid for RED, GAIL-b and SAIL-b in Fig. 2. The figure shows that SAIL-b performs consistently with expert performance. Though GAIL-b appears to be only slightly worse in average performance, the degradation is caused by occasional and highly undesirable crashes, suggesting incomplete imitation of the expert. RED performs the worst in average performance, but is consistent with no failure modes detected. The suggests that the proposed method combines the advantages of both support guidance and adversarial learning. Comparing SAIL against SAIL-b, we observe that the bounded variant generally produces policies with smaller standard deviations and better performances, especially for Ant and Humanoid. This is likely due to the fact that SAIL-b receives equal contribution from both support guidance and adversarial learning, as r red and r gail have the same range in this formulation. In addition, we note that GAIL fails to imitate the expert in Ant, while GAIL-b performs significantly better. The suggest that restricting the range of the adversarial reward could improve performance. To assess the sensitivity with respect to random seeds, we plot the training progress against number of iterations for the evaluated algorithms in Fig. 4, Each iteration consists of 1000 environment steps. The figure reports mean and standard deviation of each algorithm, across the 5 random seeds. Fig. 4 shows that SAIL-b is more sample efficient and stable in Reacher, Ant and Humanoid tasks; and is comparable to the other algorithms in the remaining tasks. Consistent with our analysis in Section 3.3, SAIL-b appears at least as efficient as GAIL even when the support guidance (i.e., the performance of RED) suffers from insufficient expert data in Hopper, HalfCheetah and Walker2d. In Reacher, Ant and Humanoid, SAIL-b benefits from the support guidance and achieves better performance and training stability. In particular, we note that without support guidance, GAIL fails to imitate the expert in Ant (Fig. 4e). Similar failures were also observed in. GAIL is also more sensitive to initial conditions: in Humanoid, GAIL converged to sub-optimal policies in 2 out 5 seeds. Lastly, while RED improves noticeably faster during early training in Humanoid, it converged to a sub-optimal policy eventually. In this paper, we propose Support-guided Adversarial Imitation Learning by combining support guidance with adversarial imitation learning. Our approach is complementary to existing adversarial imitation learning algorithms, and addresses several challenges associated with them. More broadly, our show that expert demonstrations contain rich sources of information for imitation learning. Effectively combining different sources of reinforcement learning signals from the expert demonstrations produces more efficient and stable algorithms by constraining the policy search space; and appears to be a promising direction for future research. 10413.1 ± 47.0 RED and SAIL use for support estimation. We use the default networks from RED 4. We set σ following the heuristic in that (s, a) from the expert trajectories mostly have reward close to 1. For fair comparisons, all algorithms shared hyperparameters for each task. We present them in the table below, including discriminator learning rate l D, discount factor γ, number of policy steps per iteration n G, and whether the policy has fixed variance. All other hyperparameters are set to their default values from OpenAI's baselines. To compare our method with the technique of introducing virtual absorbing state (AS) , we also construct a goal-terminal environment where the only terminal state is successful landing at the goal, because the AS technique cannot be directly applied in the no-terminal environment. We present the in Appendix C. The suggest that AS overall improves both the mean performance and standard deviations for both GAIL and SAIL. Specifically, the technique is able to mitigates the survival bias in GAIL significantly. However, SAIL still compares favorably to the technique in the goal-terminal environment. Further, since AS and support guidance is not mutually exclusive, we also combine them and report the performances. The suggest that support guidance is compatible with AS, and achieves overall the best performance with low standard deviations. The also suggest that both AS and support guidance partially mitigate the reward bias, but don't fully solve it. We will further explore this issue in future work.

represent this text to find a much much longer article about it with say hundreds-thousands of words:

We unify support estimation with the family of Adversarial Imitation Learning algorithms into Support-guided Adversarial Imitation Learning, a more robust and stable imitation learning framework.

represent the short summary for finding the corresponding scientific article (long)

We propose Dual Actor-Critic algorithm, which is derived in a principled way from the Lagrangian dual form of the Bellman optimality equation. The algorithm achieves the state-of-the-art performances across several benchmarks.


Question: Represent

Question: We consider the task of few shot link prediction, where the goal is to predict missing edges across multiple graphs using only a small sample of known edges. We show that current link prediction methods are generally ill-equipped to handle this task---as they cannot effectively transfer knowledge between graphs in a multi-graph setting and are unable to effectively learn from very sparse data. To address this challenge, we introduce a new gradient-based meta learning framework, Meta-Graph, that leverages higher-order gradients along with a learned graph signature function that conditionally generates a graph neural network initialization. Using a novel set of few shot link prediction benchmarks, we show that Meta-Graph enables not only fast adaptation but also better final convergence and can effectively learn using only a small sample of true edges. Given a graph representing known relationships between a set of nodes, the goal of link prediction is to learn from the graph and infer novel or previously unknown relationships . For instance, in a social network we may use link prediction to power a friendship recommendation system , or in the case of biological network data we might use link prediction to infer possible relationships between drugs, proteins, and diseases . However, despite its popularity, previous work on link prediction generally focuses only on one particular problem setting: it generally assumes that link prediction is to be performed on a single large graph and that this graph is relatively complete, i.e., that at least 50% of the true edges are observed during training (e.g., see ; b; ; Lü &). In this work, we consider the more challenging setting of few shot link prediction, where the goal is to perform link prediction on multiple graphs that contain only a small fraction of their true, underlying edges. This task is inspired by applications where we have access to multiple graphs from a single domain but where each of these individual graphs contains only a small fraction of the true, underlying edges. For example, in the biological setting, high-throughput interactomics offers the possibility to estimate thousands of biological interaction networks from different tissues, cell types, and organisms ; however, these estimated relationships can be noisy and sparse, and we need learning algorithms that can leverage information across these multiple graphs in order to overcome this sparsity. Similarly, in the e-commerce and social network settings, link prediction can often have a large impact in cases where we must quickly make predictions on sparsely-estimated graphs, such as when a service has been recently deployed to a new locale. That is to say to link prediction for a new sparse graph can benefit from transferring knowledge from other, possibly more dense, graphs assuming there is exploitable shared structure. We term this problem of link prediction from sparsely-estimated multi-graph data as few shot link prediction analogous to the popular few shot classification setting (; ;). The goal of few shot link prediction is to observe many examples of graphs from a particular domain and leverage this experience to enable fast adaptation and higher accuracy when predicting edges on a new, sparsely-estimated graph from the same domain-a task that can can also be viewed as a form of meta learning, or learning to learn (; 1992; ;) in the context of link prediction. This few shot link prediction setting is particularly challenging as current link prediction methods are generally ill-equipped to transfer knowledge between graphs in a multi-graph setting and are also unable to effectively learn from very sparse data. Present work. We introduce a new framework called Meta-Graph for few shot link prediction and also introduce a series of benchmarks for this task. We adapt the classical gradient-based metalearning formulation for few shot classification (; ;) to the graph domain. Specifically, we consider a distribution over graphs as the distribution over tasks from which a global set of parameters are learnt, and we deploy this strategy to train graph neural networks (GNNs) that are capable of few-shot link prediction. To further bootstrap fast adaptation to new graphs we also introduce a graph signature function, which learns how to map the structure of an input graph to an effective initialization point for a GNN link prediction model. We experimentally validate our approach on three link prediction benchmarks. We find that our MetaGraph approach not only achieves fast adaptation but also converges to a better overall solution in many experimental settings, with an average improvement of 5.3% in AUC at convergence over non-meta learning baselines. The basic set-up for few shot link prediction is as follows: We assume that we have a distribution p(G) over graphs, from which we can sample training graphs G i ∼ p(G), where each is defined by a set of nodes V i, edges E i, and matrix of real-valued node attributes X ∈ R |Vi|×d. When convenient, we will also equivalently represent a graph as |Vi|×|Vi| is an adjacency matrix representation of the edges in E i. We assume that each of these sampled graphs, G i, is a simple graph (i.e., contain a single type of relation and no self loops) and that every node v ∈ V i in the graph is associated with a real valued attribute vector x v ∈ R d from a common vector space. We further assume that for each graph G i we have access to only a sparse subset of the true edges In terms of distributional assumptions we assume that this p(G) is defined over a set of related graphs (e.g., graphs drawn from a common domain or application setting). Our goal is to learn a global or meta link prediction model from a set of sampled training graphs G i ∼ p(G), i = 1...n, such that we can use this meta model to quickly learn an effective link prediction model on a newly sampled graph G * ∼ p(G). More specifically, we wish to optimize a global set of parameters θ, as well as a graph signature function ψ(G i), which can be used together to generate an effective parameter initialization, φ i, for a local link prediction model on graph G i. Relationship to standard link prediction. Few shot link prediction differs from standard link prediction in three important ways: 1. Rather than learning from a single graph G, we are learning from multiple graphs {G 1, ..., G n} sampled from a common distribution or domain. 2. We presume access to only a very sparse sample of true edges. Concretely, we focus on settings where at most 30% of the edges in E i are observed during training, i.e., where By "true edges" we mean the full set of ground truth edges available in a particular dataset. 3. We distinguish between the global parameters, which are used to encode knowledge about the underlying distribution of graphs, and the local parameters φ i, which are optimized to perform link prediction on a specific graph G i. This distinction allows us to consider leveraging information from multiple graphs, while still allowing for individually-tuned link prediction models on each specific graph. Relationship to traditional meta learning. Traditional meta learning for few-shot classification, generally assumes a distribution p(T) over classification tasks, with the goal of learning global parameters that can facilitate fast adaptation to a newly sampled task T i ∼ p(T) with few examples. We instead consider a distribution p(G) over graphs with the goal of performing link prediction on a newly sampled graph. An important complication of this graph setting is that the individual predictions for each graph (i.e., the training edges) are not i.i.d.. Furthermore, for few shot link prediction we require training samples as a sparse subset of true edges that represents a small percentage of all edges in a graph. Note that for very small percentages we effectively break all graph structure and recover the supervised setting for few shot classification and thus simplifying the problem. We now outline our proposed approach, Meta-Graph, to the few shot link prediction problem. We first describe how we define the local link prediction models, which are used to perform link prediction on each specific graph G i. Next, we discuss our novel gradient-based meta learning approach to define a global model that can learn from multiple graphs to generate effective parameter initializations for the local models. The key idea behind Meta-Graph is that we use gradient-based meta learning to optimize a shared parameter initialization θ for the local models, while also learning a parametric encoding of each graph G i that can be used to modulate this parameter initialization in a graph-specific way (Figure 1). In principle, our framework can be combined with a wide variety of GNN-based link prediction approaches, but here we focus on variational graph autoencoders (VGAEs) (b) as our base link prediction framework. Formally, given a graph G = (V, A, X), the VGAE learns an inference model, q φ, that defines a distribution over node embeddings q φ (Z|A, X), where each row z v ∈ R d of Z ∈ R |V|×d is a node embedding that can be used to score the likelihood of an edge existing between pairs of nodes. The parameters of the inference model are shared across all the nodes in G, to define the approximate posterior, where the parameters of the normal distribution are learned via GNNs: and The generative component of the VGAE is then defined as i.e., the likelihood of an edge existing between two nodes, u and v, is proportional to the dot product of their node embeddings. Given the above components, the inference GNNs can be trained to minimize the variational lower bound on the training data: where a Gaussian prior is used for p(z). We build upon VGAEs due to their strong performance on standard link prediction benchmarks (b), as well as the fact that they have a well-defined probabilistic interpretation that generalizes many embedding-based approaches to link prediction (e.g., node2vec ). We describe the specific GNN implementations we deploy for the inference model in Section 3.3. The key idea behind Meta-Graph is that we use gradient-based meta learning to optimize a shared parameter initialization θ for the inference models of a VGAE, while also learning a parametric encoding ψ(G i) that modulates this parameter initialization in a graph-specific way. Specifically, given a sampled training graph G i, we initialize the inference model q φi for a VGAE link prediction model using a combination of two learned components: • A global initialization, θ, that is used to initialize all the parameters of the GNNs in the inference model. The global parameters θ are optimized via second-order gradient descent to provide an effective initialization point for any graph sampled from the distribution p(G). • A graph signature s Gi = ψ(G i) that is used to modulate the parameters of inference model φ i based on the history of observed training graphs. In particular, we assume that the inference model q φi for each graph G i can be conditioned on the graph signature. That is, we augment the inference model to q φi (Z|A, X, s Gi), where we also include the graph signature s Gi as a conditioning input. We use a k-layer graph convolutional network (GCN) (a), with sum pooling to compute the signature: where GCN denotes a k-layer GCN (as defined in (a) ), MLP denotes a densely-connected neural network, and we are summing over the node embeddings z v output from the GCN. As with the global parameters θ, the graph signature model ψ is optimized via second-order gradient descent. The overall Meta-Graph architecture is detailed in Figure 1 and the core learning algorithm is summarized in the algorithm block below. Result: Global parameters θ, Graph signature function ψ Initialize learning rates: α, Sample a mini-batch of graphs, G batch from p(G); The basic idea behind the algorithm is that we (i) sample a batch of training graphs, (ii) initialize VGAE link prediction models for these training graphs using our global parameters and signature function, (iii) run K steps of gradient descent to optimize each of these VGAE models, and (iv) use second order gradient descent to update the global parameters and signature function based on a held-out validation set of edges. As depicted in Fig 1, this corresponds to updating the GCN based encoder for the local link prediction parameters φ j and global parameters θ along with the graph signature function ψ using second order gradients. Note that since we are running K steps of gradient descent within the inner loop of Algorithm 1, we are also "meta" optimizing for fast adaptation, as θ and ψ are being trained via second-order gradient descent to optimize the local model performance after K gradient updates, where generally K ∈ {0, 1, . . ., 5}. We consider several concrete instantiations of the Meta-Graph framework, which differ in terms of how the output of the graph signature function is used to modulate the parameters of the VGAE inference models. For all the Meta-Graph variants, we build upon the standard GCN propagation rule (a) to construct the VGAE inference models. In particular, we assume that all the inference GNNs (Equation 1) are defined by stacking K neural message passing layers of the form: where h v ∈ R d denotes the embedding of node v at layer k of the model, N (v) = {u ∈ V : e u,v ∈ E} denotes the nodes in the graph neighborhood of v, and W (k) ∈ R d×d is a trainable weight matrix for layer k. The key difference between Equation 5 and the standard GCN propagation rule is that we add the modulation function m s G, which is used to modulate the message passing based on the graph signature s G = ψ(G). We describe different variations of this modulation below. In all cases, the intuition behind this modulation is that we want to compute a structural signature from the input graphs that can be used to condition the initialization of the local link prediction models. Intuitively, we expect this graph signature to encode structural properties of sampled graphs G i ∼ p(G) in order to modulate the parameters of the local VGAE link prediction models and adapt it to the current graph., we experiment with basic feature-wise linear modulation to define the modulation function m s G: Here, we restrict the modulation terms β k and γ k output by the signature function to be in [−1, 1] by applying a tanh non-linearity after Equation 4. GS-Gating. Feature-wise linear modulation of the GCN parameters (Equation 6) is an intuitive and simple choice that provides flexible modulation while still being relatively constrained. However, one drawback of the basic linear modulation is that it is "always on", and there may be instances where the modulation could actually be counter-productive to learning. To allow the model to adaptively learn when to apply modulation, we extend the feature-wise linear modulation using a sigmoid gating term, ρ k (with entries), that gates in the influence of γ and β: GS-Weights. In the final variant of Meta-Graph, we extend the gating and modulation idea by separately aggregating graph neighborhood information with and without modulation and then merging these two signals via a convex combination: where we use the basic linear modulation (Equation 6) to define m s β k,γ k. Note that a simplification of Meta-Graph, where the graph signature function is removed, can be viewed as an adaptation of model agnostic meta learning (MAML) to the few shot link prediction setting. As discussed in Section 2, there are important differences in the setup for few shot link prediction, compared to traditional few shot classification. Nonetheless, the core idea of leveraging an inner and outer loop of training in Algorithm 1-as well as using second order gradients to optimize the global parameters-can be viewed as an adaptation of MAML to the graph setting, and we provide comparisons to this simplified MAML approach in the experiments below. We formalize the key differences by depicting the graphical model of MAML as first depicted in and contrasting it with the graphical model for Meta-Graph, in Figure 1. MAML when reinterpreted for a distribution over graphs, maximizes the likelihood over all edges in the distribution. On the other hand, Meta-Graph when recast in a hierarchical Bayesian framework adds a graph signature function that influencesφ j to produce the modulated parameters φ j from N sampled edges. This explicit influence of ψ is captured by the term p(φ j |ψ, φ j) in Equation 7 below: For computational tractability we take the likelihood of the modulated parameters as a point estimate -i.e., p(φ j |ψ,φ j) = δ(ψ ·φ j). We design three novel benchmarks for the few-shot link prediction task. All of these benchmarks contain a set of graphs drawn from a common domain. In all settings, we use 80% of these graphs for training and 10% as validation graphs, where these training and validation graphs are used to optimize the global model parameters (for Meta-Graph) or pre-train weights (for various baseline approaches). We then provide the remaining 10% of the graphs as test graphs, and our goal is to fine-tune or train a model on these test graphs to achieve high link prediction accuracy. Note that in this few shot link prediction setting, there are train/val/test splits at both the level of graphs and edges: for every individual graph, we are optimizing a model using the training edges to predict the likelihood of the test edges, but we are also training on multiple graphs with the goal of facilitating fast adaptation to new graphs via the global model parameters. Our goal is to use our benchmarks to investigate four key empirical questions: Q1 How does the overall performance of Meta-Graph compare to various baselines, including (i) a simple adaptation of MAML (i.e., an ablation of Meta-Graph where the graph signature function is removed), (ii), standard pre-training approaches where we pre-train the VGAE model on the training graphs before fine-tuning on the test graphs, and (iii) naive baselines that do not leverage multi-graph information (i.e., a basic VGAE without pre-training, the Adamic-Adar heuristic , and DeepWalk )? Q2 How well does Meta-Graph perform in terms of fast adaption? Is Meta-Graph able to achieve strong performance after only a small number of gradient steps on the test graphs? Q3 How necessary is the graph signature function for strong performance, and how do the different variants of the Meta-Graph signature function compare across the various benchmark settings? Q4 What is learned by the graph signature function? For example, do the learned graph signatures correlate with the structural properties of the input graphs, or are they more sensitive to node feature information? Datasets. Two of our benchmarks are derived from standard multi-graph datasets from proteinprotein interaction (PPI) networks and 3D point cloud data (FirstMM-DB) . These benchmarks are traditionally used for node and graph classification, respectively, but we adapt them for link prediction. We also create a novel multi-graph dataset based upon the AMINER citation data , where each node corresponds to a paper and links represent citations. We construct individual graphs from AMINER data by sampling ego networks around nodes and create node features using embeddings of the paper abstracts (see Appendix for details). We preprocess all graphs in each domain such that each graph contains a minimum of 100 nodes and up to a maximum of 20000 nodes. For all datasets, we perform link prediction by training on a small subset (i.e., a percentage) of the edges and then attempting to predict the unseen edges (with 20% of the held-out edges used for validation). Key dataset statistics are summarized in Table 1. Baseline details. Several baselines correspond to modifications or ablations of Meta-Graph, including the straightforward adaptation of MAML (which we term MAML in the ), a finetune baseline where we pre-train a VGAE on the training graphs observed in a sequential order and finetune on the test graphs (termed Finetune). We also consider a VGAE trained individually on each test graph (termed No Finetune). For Meta-Graph and all of these baselines we employ Bayesian optimization with Thompson sampling to perform hyperparameter selection using the validation sets. We use the recommended default hyperparameters for DeepWalk and Adamic-Adar baseline is hyperparameter-free. Q1: Overall Performance. Table 2 shows the link prediction AUC for Meta-Graph and the baseline models when trained to convergence using 10%, 20% or 30% of the graph edges. In this setting, we adapt the link prediction models on the test graphs until learning converges, as determined by performance on the validation set of edges, and we report the average link prediction AUC over the test edges of the test graphs. Overall, we find that Meta-Graph achieves the highest average AUC in all but one setting, with an average relative improvement of 4.8% in AUC compared to the MAML approach and an improvement of 5.3% compared to the Finetune baseline. Notably, MetaGraph is able to maintain especially strong performance when using only 10% of the graph edges for training, highlighting how our framework can learn from very sparse samples of edges. Interestingly, in the Ego-AMINER dataset, unlike PPI and FIRSTMM DB, we observe the relative difference in performance between Meta-Graph and MAML to increase with density of the training set. We hypothesize that this is due to fickle nature of optimization with higher order gradients in MAML which is somewhat alleviated in GS-gating due to the gating mechanism. With respect to computational complexity we observe a slight overhead when comparing MetaGraph to MAML which can be reconciled by realizing that the graph signature function is not updated in the inner loop update but only in outer loop. In the Appendix, we provide additional when using larger sets of training edges, and, as expected, we find that the relative gains of Meta-Graph decrease as more and more training edges are available. Q2: Fast Adaptation. Table 3 setting we only compare to the MAML, Finetune, and No Finetune baselines, as fast adaption in this setting is not well defined for the DeepWalk and Adamic-Adar baselines. In terms of fast adaptation, we again find that Meta-Graph is able to outperform all the baselines in all but one setting, with an average relative improvement of 9.4% compared to MAML and 8.0% compared to the Finetune baseline-highlighting that Meta-Graph can not only learn from sparse samples of edges but is also able to quickly learn on new data using only a small number of gradient steps. Also, we observe poor performance for MAML in the Ego-AMINER dataset dataset which we hypothesize is due to extremely low learning rates -i.e. 1e − 7 needed for any learning, the addition of a graph signature alleviates this problem. Figure 2 shows the learning curves for the various models on the PPI and FirstMM DB datasets, where we can see that Meta-Graph learns very quickly but can also begin to overfit after only a small number of gradient updates, making early stopping essential. Q3: Choice of Meta-Graph Architecture. We study the impact of the graph signature function and its variants GS-Gating and GS-Weights by performing an ablation study using the FirstMM DB dataset. Figure 3 shows the performance of the different model variants and baselines considered as the training progresses. In addition to models that utilize different signature functions we report a random baseline where parameters are initialized but never updated allowing us to assess the inherent power of the VGAE model for few-shot link prediction. To better understand the utility of using a GCN based inference network we also report a VGAE model that uses a simple MLP on the node features and is trained analogously to Meta-Graph as a baseline. As shown in Figure 3 many versions of the signature function start at a better initialization point or quickly achieve higher AUC scores in comparison to MAML and the other baselines, but simple modulation and GS-Gating are superior to GS-Weights after a few gradient steps. Q4: What is learned by the graph signature? To gain further insight into what knowledge is transferable among graphs we use the FirstMM DB and Ego-AMINER datasets to probe and compare the output of the signature function with various graph heuristics. In particular, we treat the output of s G = ψ(G) as a vector and compute the cosine similarity between all pairs of graph in the training set (i.e., we compute the pairwise cosine similarites between graph signatures, s G). We similarly compute three pairwise graph statistics-namely, the cosine similarity between average node features in the graphs, the difference in number of nodes, and the difference in number of edges-and we compute the Pearson correlation between the pairwise graph signature similarities and these other pairwise statistics. As shown in Table 4 we find strong positive correlation in terms of Pearson correlation coefficient between node features and the output of the signature function for both datasets, indicating that the graph signature function is highly sensitive to feature information. This observation is not entirely surprising given that we use such sparse samples of edges-meaning that many structural graph properties are likely lost and making the meta-learning heavily reliant on node feature information. We also observe moderate negative correlation with respect to the average Table 4: Pearson scores between graph signature output and other graph statistics. difference in nodes and edges between pairs of graphs for FirstMM DB dataset. For Ego-AMINER we observe small positive correlation for difference in nodes and edges. We now briefly highlight related work on link prediction, meta-learning, few-shot classification, and few-shot learning in knowledge graphs. Link prediction considers the problem of predicting missing edges between two nodes in a graph that are likely to have an edge. . Common successful applications of link prediction include friend and content recommendations , shopping and movie recommendation , knowledge graph completion and even important social causes such as identifying criminals based on past activities . Historically, link prediction methods have utilized topological graph features such as common neighbors yielding strong baselines like Adamic/Adar measure , Jaccard Index among others. Other approaches include Matrix Factorization and more recently deep learning and graph neural networks based approaches (; ;) have risen to prominence. A commonality among all the above approaches is that the link prediction problem is define over a single dense graph where the objective is to predict unknown/future links within the same graph. Unlike these previous approaches, our approach considers link prediction tasks over multiple sparse graphs which are drawn from distribution over graphs akin to real world scenario such as protein-protein interaction graphs, 3D point cloud data and citation graphs in different communities. In meta-learning or learning to learn (; 1992; ;), the objective is to learn from prior experiences to form inductive biases for fast adaptation to unseen tasks. Meta-learning has been particularly effective in few-shot learning tasks with a few notable approaches broadly classified into metric based approaches (; ;), augmented memory (; ;) and optimization based approaches . Recently, there are several works that lie at the intersection of meta-learning for few-shot classification and graph based learning. learn a graph between tasks in embedding space while introduce a message propagation rule between prototypes of classes. However, both these methods are restricted to the image domain and do not consider meta-learning over a distribution of graphs as done here. Another related line of work considers the task of few-shot relation prediction in knowledge graphs. developed the first method for this task, which leverages a learned matching met-ric using both a learned embedding and one-hop graph structures. More recently introduce Meta Relational Learning framework (MetaR) that seeks to transfer relation-specific meta information to new relation types in the knowledge graph. A key distinction between few-shot relation setting and the one which we consider in this work is that we assume a distribution over graphs while in the knowledge graph setting there is only a single graph and the challenge is generalizing to new types of relations within this graph. We introduce the problem of few-shot link prediction-where the goal is to learn from multiple graph datasets to perform link prediction using small samples of graph data-and we develop the Meta-Graph framework to address this task. Our framework adapts gradient-based meta learning to optimize a shared parameter initialization for local link prediction models, while also learning a parametric encoding, or signature, of each graph, which can be used to modulate this parameter initialization in a graph-specific way. Empirically, we observed substantial gains using Meta-Graph compared to strong baselines on three distinct few-shot link prediction benchmarks. In terms of limitations and directions for future work, one key limitation is that our graph signature function is limited to modulating the local link prediction model through an encoding of the current graph, which does not explicitly capture the pairwise similarity between graphs in the dataset. Extending Meta-Graph by learning a similarity metric or kernel between graphs-which could then be used to condition meta-learning-is a natural direction for future work. Another interesting direction for future work is extending the Meta-Graph approach to multi-relational data, and exploiting similarities between relation types through a suitable Graph Signature function. To construct the Ego-Aminer dataset we first create citation graphs from different fields of study. We then select the top 100 graphs in terms number of nodes for further pre-processing. Specifically, we take the 5-core of each graph ensuring that each node has a minimum of 5-edges. We then construct ego networks by randomly sampling a node from the 5-core graph and taking its two hop neighborhood. Finally, we remove graphs with fewer than 100 nodes and greater than 20000 nodes which leads to a total of 72 graphs as reported in Table 1. We list out complete when using larger sets of training edges for PPI, FIRSTMM DB and Ego-Aminer datasets. We show the for two metrics i.e. Average AUC across all test graphs. As expected, we find that the relative gains of Meta-Graph decrease as more and more training edges are available. Table 6: 5-gradient update AUC for PPI for training edge splits

Represent the short summary for finding the corresponding scientific article (long)

We apply gradient based meta-learning to the graph domain and introduce a new graph specific transfer function to further bootstrap the process.

Represent the TLDR for finding the corresponding scientific article

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.