Question: Represent the natural language

Question: The field of few-shot learning has recently seen substantial advancements. Most of these advancements came from casting few-shot learning as a meta-learning problem. Model Agnostic Meta Learning or MAML is currently one of the best approaches for few-shot learning via meta-learning. MAML is simple, elegant and very powerful, however, it has a variety of issues, such as being very sensitive to neural network architectures, often leading to instability during training, requiring arduous hyperparameter searches to stabilize training and achieve high generalization and being very computationally expensive at both training and inference times. In this paper, we propose various modifications to MAML that not only stabilize the system, but also substantially improve the generalization performance, convergence speed and computational overhead of MAML, which we call MAML++. The human capacity to learn new concepts using only a handful of samples is immense. In stark contrast, modern deep neural networks need, at a minimum, thousands of samples before they begin to learn representations that can generalize well to unseen data-points BID11 BID9, and mostly fail when the data available is scarce. The fact that standard deep neural networks fail in the small data regime can provide hints about some of their potential shortcomings. Solving those shortcomings has the potential to open the door to understanding intelligence and advancing Artificial Intelligence. Few-shot learning encapsulates a family of methods that can learn new concepts with only a handful of data-points (usually 1-5 samples per concept). This possibility is attractive for a number of reasons. First, few-shot learning would reduce the need for data collection and labelling, thus reducing the time and resources needed to build robust machine learning models. Second, it would potentially reduce training and fine-tuning times for adapting systems to newly acquired data. Third, in many real-world problems there are only a few samples available per class and the collection of additional data is either remarkably time-consuming and costly or altogether impossible, thus necessitating the need to learn from the available few samples. The nature of few-shot learning makes it a very hard problem if no prior knowledge exists about the task at hand. For a model to be able to learn a robust model from a few samples, knowledge transfer (see e.g. BID4 from other similar tasks is key. However, manual knowledge transfer from one task to another for the purpose of fine-tuning on a new task can be a time consuming and ultimately inefficient process. Meta-learning BID22 BID24, or learning to learn BID23, can instead be used to automatically learn across-task knowledge usually referred to as across-task (or sometimes slow) knowledge such that our model can, at inference time, quickly acquire task-specific (or fast) knowledge from new tasks using only a few samples. Meta-learning can be broadly defined as a class of machine learning models that become more proficient at learning with more experience, thus learning how to learn. More specifically meta-learning involves learning at two levels. At the task-level, where the base-model is required to acquire task-specific (fast) knowledge rapidly, and at the meta-level, where the meta-model is required to slowly learn across-task (slow) knowledge. Recent work in meta-learning has produced One can see that 2 out of 3 seeds with the original strided MAML seem to become unstable and erratic, whereas all 3 of the strided MAML++ models seem to consistently converge very fast, to much higher generalization accuracy without any stability issues.state of the art in a variety of settings BID26 BID2 BID27 BID0 BID25 BID14 BID15 BID1 BID3 BID17. The application of meta-learning in the fewshot learning setting has enabled the overwhelming majority of the current state of the art few-shot learning methods BID25 BID19 BID6 BID7 BID15 BID17. One such method, known for its simplicity and state of the art performance, is Model Agnostic Meta-Learning (MAML) BID7. In MAML, the authors propose learning an initialization for a base-model such that after applying a very small number of gradient steps with respect to a training set on the base-model, the adapted model can achieve strong generalization performance on a validation set (the validation set consists of new samples from the same classes as the training set). Relating back to the definitions of metamodel and base-model, in MAML the meta-model is effectively the initialization parameters. These parameters are used to initialize the base-model, which is then used for task-specific learning on a support set, which is then evaluated on a target set. MAML is a simple yet elegant meta-learning framework that has achieved state of the art in a number of settings. However, MAML suffers from a variety of problems which: 1) cause instability during training, 2) restrict the model's generalization performance, 3) reduce the framework's flexibility, 4) increase the system's computational overhead and 5) require that the model goes through a costly (in terms of time and computation needed) hyperparameter tuning before it can work robustly on a new task. In this paper we propose MAML++, an improved variant of the MAML framework that offers the flexibility of MAML along with many improvements, such as robust and stable training, automatic learning of the inner loop hyperparameters, greatly improved computational efficiency both during inference and training and significantly improved generalization performance. MAML++ is evaluated in the few-shot learning setting where the system is able to set a new state of the art across all established few-shot learning tasks on both Omniglot and Mini-Imagenet, performing as well as or better than all established meta learning methods on both tasks. The set-to-set few-shot learning setting BID25, neatly casts few-shot learning as a meta-learning problem. In set-to-set few-shot learning we have a number of tasks, each task is composed by a support set which is used for task-level learning, and a target set which is used for evaluating the base-model on a certain task after it has acquired task-specific (or fast) knowledge. Furthermore, all available tasks are split into 3 sets, the meta-training set, the meta-validation set and the meta-test set, used for training, validating and testing our meta-learning model respectively. Once meta-learning was shown to be an effective framework for few-shot learning and the set to set approach was introduced, further developments in few-shot learning were made in quick succession. One contribution was Matching Networks BID25. Matching networks achieve few-shot learning by learning to match target set items to support set items. More specifically, a matching network learns to match the target set items to the support set items using cosine distance and a fully differentiable embedding function. First, the support set embedding function g, parameterized as a deep neural network, embeds the support set items into embedding vectors, then the target set embedding function f embeds the target set items. Once all data-item embeddings are available, cosine distance is computed for all target set embeddings when compared to all support set embeddings. As a , for each target set item, a vector of cosine distances with respect to all support set items will be generated (with each distance's column tied to the respective support set class). Then, the softmax function is applied on the generated distance vectors, to convert them into probability distributions over the support set classes. Another notable advancement was the gradient-conditional meta-learner LSTM BID19 that learns how to update a base-learner model. At inference time, the meta-learner model applies a single update on the base-learner given gradients with respect to the support set. The fully updated base-model then computes predictions on the target set. The target set predictions are then used to compute a task loss. Furthermore they jointly learn the meta-learner's parameters as well as the base-learners initializations such that after a small number of steps it can do very well on a given task. The authors ran experiments on Mini-Imagenet where they exceed the performance of Matching Networks. In Model Agnostic meta-learning (MAML) BID7 ) the authors proposed increasing the gradient update steps on the base-model and replacing the meta-learner LSTM with Batch Stochastic Gradient Descent BID11, which as a speeds up the process of learning and interestingly improves generalization performance and achieves state of the art performance in both Omniglot and Mini-Imagenet. In Meta-SGD BID15 ) the authors proposed learning a static learning rate and an update direction for each parameter in the base-model, in addition to learning the initialization parameters of the base-model. Meta-SGD showcases significantly improved generalization performance (when compared to MAML) across all few-shot learning tasks, whilst only requiring a single inner loop update step. However this practice effectively doubles the model parameters and computational overheads of the system. Model Agnostic Meta-Learning (MAML) BID7 ) is a meta-learning framework for fewshot learning. MAML is elegantly simple yet can produce state of the art in few-shot regression/classification and reinforcement learning problems. In a sentence, MAML learns good initialization parameters for a network, such that after a few steps of standard training on a few-shot dataset, the network will perform well on that few shot task. More formally, we define the base model to be a neural network f θ with meta-parameters θ. We want to learn an initial θ = θ 0 that, after a small number N of gradient update steps on data from a support set S b to obtain θ N, the network performs well on that task's target set T b. Here b is the index of a particular support set task in a batch of support set tasks. This set of N updates steps is called the inner-loop update process. The updated base-network parameters after i steps on data from the support task S b can be expressed as: DISPLAYFORM0 where α is the learning rate, θ b i are the base-network weights after i steps towards task b, DISPLAYFORM1 ) is the loss on the support set of task b after (i − 1) (i.e. the previous step) update steps. Assuming that our task batch size is B we can define a meta-objective, which can be expressed as: DISPLAYFORM2 where we have explicitly denoted the dependence of θ b N on θ 0, given by unrolling. The objective measures the quality of an initialization θ 0 in terms of the total loss of using that initialization across all tasks. This meta objective is now minimized to optimize the initial parameter value θ 0. It is this initial θ 0 that contains the across-task knowledge. The optimization of this meta-objective is called the outer-loop update process. The ing update for the meta-parameters θ 0 can be expressed as: DISPLAYFORM3 where β is a learning rate and L T b denotes the loss on the target set for task b. In this paper we use the cross-entropy BID5 BID20 loss throughout. The simplicity, elegance and high performance of MAML make it a very powerful framework for meta-learning. However, MAML has also many issues that make it problematic to use. Training Instability: Depending on the neural network architecture and the overall hyperparameter setup, MAML can be very unstable during training as illustrated in FIG0. Optimizing the outer loop involved backpropagating derivatives through an unfolded inner loop consisting of the same network multiple times. This alone could be cause for gradient issues. However, the gradient issues are further compounded by the model architecture, which is a standard 4-layer convolutional network without skip-connections. The lack of any skip-connections means that every gradient must be passed through each convolutional layer many times; effectively the gradients will be multiplied by the same sets of parameters multiple times. After multiple back-propagation passes, the large depth structure of the unfolded network and lack of skip connections can cause gradient explosions and diminishing gradient problems respectively. Second Order Derivative Cost: Optimization through gradient update steps requires the computation of second order gradients which are very expensive to compute. The authors of MAML proposed using first-order approximations to speed up the process by a factor of three, however using these approximations can have a negative impact on the final generalization error. Further attempts at using first order methods have been attempted in Reptile BID18 where the authors apply standard SGD on a base-model and then take a step from their initialization parameters towards the parameters of the base-model after N steps. The of Reptile vary, in some cases exceeding MAML, and in others producing inferior to MAML. Approaches to reduce computation time while not sacrificing generalization performance have yet to be proposed. Absence of Batch Normalization Statistic Accumulation: A further issue that affects the generalization performance is the way that batch normalization is used in the experiments in the original MAML paper. Instead of accumulating running statistics, the statistics of the current batch were used for batch normalization. This in batch normalization being less effective, since the biases learned have to accommodate for a variety of different means and standard deviations instead of a single mean and standard deviation. On the other hand, if batch normalization uses accumulated running statistics it will eventually converge to some global mean and standard deviation. This leaves only a single mean and standard deviation to learn biases for. Using running statistics instead of batch statistics, can greatly increase convergence speed, stability and generalization performance as the normalized features will in smoother optimization landscape BID21.Shared (across step) Batch Normalization Bias: An additional problem with batch normalization in MAML stems from the fact that batch normalization biases are not updated in the inner-loop; instead the same biases are used throughout all iterations of base-models. Doing this implicitly assumes that all base-models are the same throughout the inner loop updates and hence have the same distribution of features passing through them. This is a false assumption to make, since, with each inner loop update, a new base-model is instantiated that is different enough from the previous one to be considered a new model from a bias estimation point of view. Thus learning a single set of biases for all iterations of the base-model can restrict performance. Shared Inner Loop (across step and across parameter) Learning Rate: One issue that affects both generalization and convergence speed (in terms of training iterations) is the issue of using a shared learning rate for all parameters and all update-steps. Doing so introduces two major problems. Having a fixed learning rate requires doing multiple hyperparameter searches to find the correct learning rate for a specific dataset; this process can be very computationally costly, depending on how search is done. The authors in BID15 propose to learn a learning rate and update direction for each parameter of the network. Doing so solves the issue of manually having to search for the right learning rate, and also allows individual parameters to have smaller or larger learning rates. However this approach brings its own problems. Learning a learning rate for each network parameter means increased computational effort and increased memory usage since the network contains between 40K and 50K parameters depending on the dimensionality of the data-points. Fixed Outer Loop Learning Rate: In MAML the authors use Adam with a fixed learning rate to optimize the meta-objective. Annealing the learning rate using either step or cosine functions has proven crucial to achieving state of the art generalization performance in a multitude of settings BID16 BID8 BID13 BID9. Thus, we theorize that using a static learning rate reduces MAML's generalization performance and might also be a reason for slower optimization. Furthermore, having a fixed learning rate might mean that one has to spend more (computational) time tuning the learning rate. In this section we propose methods for solving the issues with the MAML framework, described in Section 3.1. Each solution has a reference identical to the reference of the issue it is attempting to solve. Gradient Instability → Multi-Step Loss Optimization (MSL): MAML works by minimizing the target set loss computed by the base-network after it has completed all of its inner-loop updates towards a support set task. Instead we propose minimizing the target set loss computed by the base-network after every step towards a support set task. More specifically, we propose that the loss minimized is a weighted sum of the target set losses after every support set loss update. More formally: DISPLAYFORM0 Where β is a learning rate, DISPLAYFORM1 ) denotes the target set loss of task b when using the base-network weights after i steps towards minimizing the support set task and v i denotes the importance weight of the target set loss at step i, which is used to compute the weighted sum. By using the multi-step loss proposed above we improve gradient propagation, since now the basenetwork weights at every step receive gradients both directly (for the current step loss) and indirectly (from losses coming from subsequent steps). With the original methodology described in Section 3 the base-network weights at every step except the last one were optimized implicitly as a of backpropagation, which caused many of the instability issues MAML had. However using the multi-step loss alleviates this issue as illustrated in FIG0. Furthermore, we employ an annealed weighting for the per step losses. Initially all losses have equal contributions towards the loss, but as iterations increase, we decrease the contributions from earlier steps and slowly increase the contribution of later steps. This is done to ensure that as training progresses the final step loss receives more attention from the optimizer thus ensuring it reaches the lowest possible loss. If the annealing is not used, we found that the final loss might be higher than with the original formulation. Second Order Derivative Cost → Derivative-Order Annealing (DA): One way of making MAML more computationally efficient is reducing the number of inner-loop updates needed, which can be achieved with some of the methods described in subsequent sections of this report. However, in this paragraph, we propose a method that reduces the per-step computational overhead directly. The authors of MAML proposed the usage of first-order approximations of the gradient derivatives. However they applied the first-order approximation throughout the whole of the training phase. Instead, we propose to anneal the derivative-order as training progresses. More specifically, we propose to use first-order gradients for the first 50 epochs of the training phase, and to then switch to second-order gradients for the remainder of the training phase. We empirically demonstrate that doing so greatly speeds up the first 50 epochs, while allowing the second-order training needed to achieve the strong generalization performance the second-order gradients provide to the model. An additional interesting observation is that derivative-order annealing experiments showed no incidents of exploding or diminishing gradients, contrary to second-order only experiments which were more unstable. Using first-order before starting to use second-order derivatives can be used as a strong pretraining method that learns parameters less likely to produce gradient explosion/diminishment issues. Absence of Batch Normalization Statistic Accumulation → Per-Step Batch Normalization Running Statistics (BNRS): In the original implementation of the authors used only the current batch statistics as the batch normalization statistics. This, we argue, caused a variety of undesirable effects described in Section 3.1. To alleviate the issues we propose using running batch statistics for batch normalization. A naive implementation of batch normalization in the context of MAML would require sharing running batch statistics across all update steps of the inner-loop fast-knowledge acquisition process. However doing so would cause the undesirable consequence that the statistics stored be shared across all inner loop updates of the network. This would cause optimization issues and potentially slow down or altogether halt optimization, due to the increasing complexity of learning parameters that can work across various updates of the network parameters. A better alternative would be to collect statistics in a per-step regime. To collect running statistics per-step, one needs to instantiate N (where N is the total number of inner-loop update steps) sets of running mean and running standard deviation for each batch normalization layer in the network and update the running statistics respectively with the steps being taken during the optimization. The per-step batch normalization methodology should speed up optimization of MAML whilst potentially improving generalization performance. In the MAML paper the authors trained their model to learn a single set of biases for each layer. Doing so assumes that the distributions of features passing through the network are similar. However, this is a false assumption since the base-model is updated for a number of times, thus making the feature distributions increasingly dissimilar from each other. To fix this problem we propose learning a set of biases per-step within the inner-loop update process. Doing so, means that batch normalization will learn biases specific to the feature distributions seen at each set, which should increase convergence speed, stability and generalization performance. Shared Inner Loop Learning Rate (across step and across parameter) → Learning Per-Layer Per-Step Learning Rates and Gradient Directions (LSLR): Previous work in BID15 demonstrated that learning a learning rate and gradient direction for each parameter in the basenetwork improved the generalization performance of the system. However, that had the consequence of increased number of parameters and increased computational overhead. So instead, we propose, learning a learning rate and direction for each layer in the network as well as learning different learning rates for each adaptation of the base-network as it takes steps. Learning a learning rate and direction for each layer instead for each parameter should reduce memory and computation needed whilst providing additional flexibility in the update steps. Furthermore, for each learning rate learned, there will be N instances of that learning rate, one for each step to be taken. By doing this, the parameters are free to learn to decrease the learning rates at each step which may help alleviate overfitting. Fixed Outer Loop Learning Rate → Cosine Annealing of Meta-Optimizer Learning Rate (CA): In MAML the authors use a static learning rate for the optimizer of the meta-model. Annealing the learning rate, either by using step-functions BID8 or cosine functions BID16 has proved vital in learning models with higher generalization power. The cosine annealing scheduling has been especially effective in producing state of the art whilst removing the need for any hyper-parameter searching on the learning rate space. Thus, we propose applying the cosine annealing scheduling on the meta-model's optimizer (i.e. the meta-optimizer). Annealing the learning rate allows the model to fit the training set more effectively and as a might produce higher generalization performance. The datasets used to evaluate our methods were the Omniglot and Mini-Imagenet BID25 BID19 datasets. Each dataset is split into 3 sets, a training, validation and test set. The Omniglot dataset is composed of 1623 characters classes from various alphabets. There exist 20 instances of each class in the dataset. For Omniglot we shuffle all character classes and randomly select 1150 for the training set and from the remaining classes we use 50 for validation and 423 for testing. In most few-shot learning papers the first 1200 classes are used for training and the remaining for testing. However, having a small validation set to choose the best model is crucial, so we choose to use a small set of 50 classes as validation set. For each class we use all available 20 samples in the sets. Furthermore for the Omniglot dataset, data augmentation is used on the images in the form of rotations of 90 degree increments. Class samples that are rotated are considered new classes, e.g. a 180 degree rotated character C is considered a different class from a non rotated C, thus effectively having 1623 x 4 classes in total. However the rotated classes are generated dynamically after the character classes have been split into the sets such that rotated samples from a class reside in the same set (i.e. the training, validation or test set). The Mini-Imagenet dataset was proposed in BID19, it consists of 600 instances of 100 classes from the ImageNet dataset, scaled down to 84x84. We use the split proposed in BID19, which consists of 64 classes for training, 12 classes for validation and 24 classes for testing. To evaluate our methods we adopted a hierarchical hyperparameter search methodology. First we began with the baseline MAML experiments, which were ran on the 5/20-way and 1/5-shot settings on the Omniglot dataset and the 5-way 1/5-shot setting on the Mini-Imagenet dataset. Then we added each one of our 6 methodologies on top of the default MAML and ran experiments for each one separately. Once this stage was completed we combined the approaches that showed improvements in either generalization performance or convergence speed (both in terms of number of epochs and clock-time) and ran a final experiment to establish any potential gains from the combination of the techniques. An experiment consisted of training for 150 epochs, each epoch consisting of 500 iterations. At the end of each epoch, we evaluated the performance of the model on the validation set. Upon completion of all epochs, an ensemble of the top 3 performing per-epoch-models on the validation set were applied on the test set, thus producing the final test performance of the model. An evaluation ran consisted of inference on 600 unique tasks. A distinction between the training and evaluation tasks, was that the training tasks were generated dynamically continually without repeating previously sampled tasks, whilst the 600 evaluation tasks generated were identical across epochs. Thus ensuring that the comparison between models was fair, from an evaluation set viewpoint. Every experiment was repeated for 3 independent runs. The models were trained using the Adam optimizer with a learning rate of 0.001, β 1 = 0.9 and β 2 = 0.99. Furthermore, all Omniglot experiments used a task batch size of 16, whereas for the Mini-Imagenet experiments we used a task batch size of 4 and 2 for the 5-way 1-shot and 5-way 5-shot experiments respectively. Our proposed methodologies are empirically shown to improve the original MAML framework. In TAB0 one can see how our proposed approach performs on Omniglot. Each proposed methodology can individually outperform MAML, however, the most notable improvements come from the learned per-step per-layer learning rates and the per-step batch normalization methodology. In the 5-way 1-shot tasks it achieves 99.47% and in the 20-way Omniglot tasks MAML++ achieves 97.76% and 99.33% in the 1-shot and 5-shot tasks respectively. MAML++ also showcases improved convergence speed in terms of training iterations required to reach the best validation performance. Furthermore, the multi-step loss optimization technique substantially improves the training stability of the model as illustrated in FIG0. In TAB0 we also include the of our own implementation of MAML, which reproduces all except the 20-way 1-shot Omniglot case. Difficulty in replicating the specific has also been noted before in BID10. We base our on the relative performance between our own MAML implementation and the proposed methodologies. TAB1 showcases MAML++on Mini-Imagenet tasks, where MAML++ sets a new state of the art in both the 5-way 1-shot and 5-shot cases where the method achieves 52.15% and 68.32% respectively. More notably, MAML++ can achieve very strong 1-shot of 51.05% with only a single inner loop step required. Not only is MAML++ cheaper due to the usage of derivative order annealing, but also because of the much reduced inner loop steps. Another notable observation is that MAML++converges to its best generalization performance much faster (in terms of iterations required) when compared to MAML as shown in FIG0. In this paper we delve deep into what makes or breaks the MAML framework and propose multiple ways to reduce the inner loop hyperparameter sensitivity, improve the generalization error, stabilize and speed up MAML. The ing approach, called MAML++sets a new state of the art across all few-shot tasks, across Omniglot and Mini-Imagenet. The of the approach indicate that learning per-step learning rates, batch normalization parameters and optimizing on per-step target losses appears to be key for fast, highly automatic and strongly generalizable few-shot learning.

MAML is great, but it has many problems, we solve many of those problems and as a result we learn most hyper parameters end to end, speed-up training and inference and set a new SOTA in few-shot learning

We identify and formalize the memorization problem in meta-learning and solve this problem with novel meta-regularization method, which greatly expand the domain that meta-learning can be  applicable to and effective on.


Question: Represent the scientific article to find a short TLDR

Question: Community detection in graphs is of central importance in graph mining, machine learning and network science. Detecting overlapping communities is especially challenging, and remains an open problem. Motivated by the success of graph-based deep learning in other graph-related tasks, we study the applicability of this framework for overlapping community detection. We propose a probabilistic model for overlapping community detection based on the graph neural network architecture. Despite its simplicity, our model outperforms the existing approaches in the community recovery task by a large margin. Moreover, due to the inductive formulation, the proposed model is able to perform out-of-sample community detection for nodes that were not present at training time Graphs provide a natural way of representing complex real-world systems. For understanding the structure and behavior of these systems, community detection methods are an essential tool. Detecting communities allows us to analyze social networks BID9, to detect fraud BID24, to discover functional units of the brain BID8, and to predict functions of proteins BID26. Over the last decades, this problem has attracted significant attention of the research community and numerous models and algorithms have been proposed BID32. In particular, it is a well known fact that communities in real graphs are in fact overlapping BID34, thus, requiring the development of advanced models to capture this complex structure. In this regard, the advent of deep learning methods for graph-structured data opens new possibilities for designing more accurate and more scalable algorithms. Indeed, deep learning on graphs has already shown state-of-the-art in s for various graph-related tasks such as semi-supervised node classification and link prediction BID2. Likewise, a few deep learning methods for community detection in graphs have been proposed BID36 BID4. However, they all have one drawback in common: they only focus on the special case of disjoint (non-overlapping) communities. Handling overlapping communities, is a requirement not yet met by existing deep learning approaches to community detection. In this paper we propose an end-to-end deep probabilistic model for overlapping community detection in graphs. Our core idea lies in predicting the community affiliations using a graph neural network. Despite its simplicity, our model achieves state-of-the art in community recovery and significantly outperforms the existing approaches. Moreover, our model is able to perform out-of-sample (inductive) community detection for nodes that were not seen at training time. To summarize, our main contributions are:• We propose the Deep Overlapping Community detection (DOC) model -a simple, yet effective deep learning model for overlapping community detection in graphs. DOC is one of few methods is able to perform community detection both transductively and inductively.• We introduce 5 new datasets for overlapping community detection, which can act as a benchmark to stimulate future work in this research area.• We perform a thorough experimental evaluation of our model, and show its superior performance when comparing with established methods for overlapping community detection. Assume that we are given an undirected unweighted graph G = (V, E), with N:= |V| nodes and M:= |E| edges, represented by a symmetric adjacency matrix A ∈ {0, 1} N ×N. Moreover, each node is associated with D real-valued attributes, that can be represented as a matrix X ∈ R N ×D. The goal of overlapping community detection is to assign nodes in the graph into C communities. Such assignment can be represented as a non-negative community affiliation matrix F ∈ R N ×C ≥0, where F uc denotes the strength of node u's membership in community c (with the notable special case of binary hard-assignment F ∈ {0, 1} N ×C ). There is no single universally definition of community in the literature. However, most recent works tend to agree with the statement that a community is a group of nodes that have a higher probability to form edges with each other than with other nodes in the graph BID6.One can broadly subdivide the existing methods for overlapping community detection into three categories: approaches based on non-negative matrix factorization (NMF), probabilistic inference, or heuristics. Methods based on NMF are trying to recover the community affiliation matrix F by performing a low-rank decomposition of the adjacency matrix A or some other related matrix BID31 BID15. Probabilistic approaches, such as or BID37, treat F as a latent variable in a generative model for the graph, p(A, F). This way the problem of community detection is cast as an instance of probabilistic inference. Lastly, heuristic-based approaches usually define a goodness measure, like within-community edge density BID7, and then directly optimize it. All of these approaches can be very generally formulated as an optimization problem min DISPLAYFORM0 for an appropriate choice of the loss function L, be it Frobenius norm L(DISPLAYFORM1 Besides these traditional approaches, one can also view the problem of community detection through the lens of representation learning. The community affiliation matrix F can be considered as an embedding of nodes into R C ≥0, with the aim of preserving the community structure. Given the recent success of deep representation learning for graphs BID2, a question arises: "Can the advances in deep representation learning for graphs be used to design better community detection algorithms?".A very simple idea is to first apply a node embedding approach to the graph, and then cluster the nodes in the embedding space using k-means to obtain communities (as done in, e.g., BID29). However, such approach is only able to detect disjoint communities, which does not correspond to the structure of communities in real-world graphs BID34. Instead, we argue that an end-to-end deep learning architecture able to detect overlapping communities is preferable. Traditional community detection methods treat F as a free variable, with respect to which optimization is performed FIG0 ). This is similar to how embeddings are learned in methods like DeepWalk BID23 and node2vec . In contrast, recent works of BID13; BID1 have adopted the approach of defining the embeddings as a function of node attributes F:= f θ (X, A) and solving the optimization problem DISPLAYFORM2 where f θ is defined by a neural network. 1 Such formulation allows to• achieve better performance in downstream tasks like link prediction and node classification;• naturally incorporate the attributes X without hand-crafting the generative model p(X, F);• generate embeddings inductively for previously unseen nodes. We propose to use this framework for overlapping community detection, and describe our model in the next section. We let the community affiliations be produced by a three-layer graph convolutional neural network (GCN), as defined in BID14. DISPLAYFORM0 whereÂ =D DISPLAYFORM1 2 is the normalized adjacency matrix,Ã = A + I, andD the corresponding degree matrix. A ReLU nonlinearity is applied element-wise to the output layer to ensure nonnegativity of the community affiliation matrix F. Any other graph neural network architecture can be used here -we choose GCN because of its simplicity and popularity. Link function. A good F explains well the community structure of the graph. To model this formally, we adopt a probabilistic approach to community detection where we need to define the likelihood p(A|F). A standard assumption in probabilistic community detection is that the edges A uv are conditionally independent given the community memberships F. Thus, once the F matrix is given, every pair of nodes (u, v) produces an interaction based on their community affiliations DISPLAYFORM2. For a probabilistic interpretation, this interaction is transformed into an edge probability by means of a link function g: R ≥0 →. The edge probability is then given by DISPLAYFORM3 We consider two choices for the link function g: Bernoulli-Poisson link and sigmoid link. Bernoulli-Poisson link, defined as ξ(X uv) = 1 − exp(−X uv), is a common probabilistic model for overlapping community detection BID37 BID28. Note, that under the BP model a pair of nodes that have no communities in common (i.e. F u F T v = 0) have a zero probability of forming an edge. This is an unrealistic assumption, which can be easily fixed by adding a small offset ε > 0, that is ξ(X uv) = 1 − exp(−X uv − ε). DISPLAYFORM4 −1, is the standard choice for binary classification problems. It can also be used to convert the edge scores into probabilities in probabilistic models for graphs BID16 BID27 BID13. Since a non-negative F implies that the interactions between every pair of nodes X uv are at least 0, the edge probability under the sigmoid model is always above σ = 0.5. This can be fixed by introducing an offset: g(X uv) = σ(X uv − b). The offset b becomes an additional variable to optimize over, closedform expression for which is provided in BID25. However, while optimizing over b produces better likelihood scores, we have empirically observed that fixing it to zero leads to the same performance in community recovery (Section 4.3). Thus, we set b = 0 in our experiments. We consider both link functions, and denote the two variants of our model as DOC-BP and DOCSigmoid for Bernoulli-Poisson and sigmoid link functions respectively. Maximizing the likelihood is equivalent to minimizing the negative log-likelihood, which corresponds to the well-known binary cross-entropy loss function. Since real-world graphs are extremely sparse (only 10 −2 − 10 −5 of possible edges are present), we are dealing with an extremely imbalanced binary classification problem. A standard way of dealing with this problem is by balancing the contribution from both classes, which corresponds to the following objective function where P E and P N stand for uniform distributions over edges and non-edges respectively. DISPLAYFORM0 Evaluating the gradient of the full loss requires O(N 2) operations (since we need to compute the expectation over N 2 possible edges/non-edges). This is impractical even for moderately-sized graphs. Instead, we optimize the objective using stochastic gradient descent. That is, at every iteration we approximate ∇L using S randomly sampled edges, as well as the same number of non-edges. To summarize, we use stochastic gradient descent to optimize the objective DISPLAYFORM1 where the parameters θ are the weights of the neural network, Datasets. We perform all our experiments using the following real-world graph datasets. Facebook BID18 ) is a collection of small (100-1000 nodes) ego-networks from the Facebook graph. In our experiments we consider the 5 largest of these ego-networks (Facebook-0, Facebook-107, Facebook-1684, Facebook-1912 . DISPLAYFORM2 Larger graph datasets (1000+ nodes) with reliable ground-truth overlapping community information, and node attributes are not openly available, which hampers the evaluation of methods for overlapping community detection in attributed graphs. For this reason we have collected and preprocessed 5 real-world datasets, that satisfy these criteria and can act as future benchmarks (we will provide the datasets for download after the blind-reviewing phase). Coauthor-CS and CoauthorPhysics are subsets of the Microsoft Academic co-authorship graph, constructed based on the data from the KDD Cup 2016 2. Communities correspond to research areas in computer science and physics respectively. Reddit-Technology and Reddit-Gaming represent user-user graphs from the content-sharing platform Reddit 3. Communities correspond to subreddits -topic-specific communities that users participate in. Amazon is a segment of the Amazon co-purchase graph BID19, where product categories represent the communities. Details about how the datasets were constructed and exploratory analysis are provided in Appendix B.Model architecture. We denote the model variant with the Bernoulli-Poisson link as DOC-BP, and the model variant with the sigmoid link as DOC-Sigmoid. For all experiments we use a 3-layer GCN (Equation 3) as the basis for both models. We use the same model configuration for all other experiments, unless otherwise specified. More details about the model and the training procedure are provided in Appendix A. All reported are averaged over 10 random initializations, unless otherwise specified. As mentioned in Section 3, evaluation of the full loss (Equation 5) and its gradients is computationally prohibitive due to its O(N 2) scaling. Instead, we propose to use a stochastic approximation, that only depends on the fixed batch size S. We perform the following experiment to ensure that our training procedure converges to the same , as when using the full objective. Experimental setup. We train the the two variants of the model on the Facebook-1912 dataset, since it is small enough (N = 755) for full-batch training to be feasible. We compare the full-batch training procedure with stochastic training for different choices of the batch size S. Starting with the same initialization, we measure the respective full losses (Equation 5) over the iterations. Results. FIG0 shows training curves for batch sizes S ∈ {500, 1000, 2500, 4000, 5000}, as well as for full-batch training. As we see, the stochastic training procedure is stable. For all batch sizes the loss converges very closely to the value achieved by full-batch training. The standard way of comparing overlapping community detection algorithms is by assessing how well they can recover communities in graphs, for which the ground truth community affiliations are known. It may happen that the information used as "ground truth communities" does not correlate with the graph structure BID22. For the datasets considered in this paper, however, ground truth communities make sense both intuitively and quantitatively (see Appendix B for a more detailed discussion). Therefore, good performance in this experiment is a good indicator of the utility of an algorithm. Predicting community membership. In order to compare the detected communities to the ground truth, we first need to convert continuous community affiliations F into binary community assignments. We assign node u to community c if F uc is above a threshold ρ. We set ρ = 0.4 for DOC-BP and ρ = 0.2 for DOC-Sigmoid, as these are the values that achieve the best performance on the Facebook-1912 dataset. Metrics. We use overlapping normalized mutual information (NMI), as defined by BID20, in order to quantify the agreement of the detected communities with the ground-truth data. Baselines. We compare our method against a number of established methods for overlapping community detection. BigCLAM ) is a probabilistic model based on the Bernoulli-Poisson link that only considers the graph structure. CESNA is an extension of BigCLAM, that additionally models the generative process for node attributes. SNMF BID15 and CDE BID17 are non-negative matrix factorization approaches for overlapping community detection. We also compared against the LDense algorithm from BID7 -a heuristic-based approach, that finds communities with maximum edge density and similar attributes. However, since it achieved less than 1% NMI for 8 out of 10 datasets, we don't include the for LDense into the table. To ensure a fair comparison, all methods were given the true number of communities C. Other hyperparameters were set to their recommended values. Detailed configurations of the baselines are provided in Appendix C.Results. TAB1 shows how well different methods score in recovery of ground-truth communities. DOC-BP achieves the best or the second best score for 9 out of 10 datasets. DOC-Sigmoid achieves the best or the second best score 10 out of 10 times. This demonstrates the potential of deep learning methods for overlapping community detection. CESNA could not be run for the Amazon dataset, because it cannot handle continuous attributes. In contrast, both DOC model variants can be used with any kind of attributes out of the box. CDE was not able to process any of the graphs with N ≥ 7K nodes within 24 hours. On the other hand, both DOC-BP and DOC-Sigmoid converged in 30s-6min for all datasets except Amazon, where it took up to 20 minutes because of the dense attribute matrix. As we just saw, the DOC-BP and DOC-Sigmoid models, both based on the GCN architecture, are able to achieve superior performance in community detection. Intuitively, it makes sense to use a graph neural network (GNN) in our setting, since it allows to incorporate the attribute information and also produces similar community vectors, F u, for adjacent nodes. Nevertheless, we should ask whether it's possible achieve comparable with a simpler model. To answer this question, we consider the following two baselines. ), we use a simple fully-connected neural network to generate F. DISPLAYFORM0 This is indeed related to the model proposed by BID12. For this baseline, we use the same configuration (number and sizes of layers, training procedure, etc.) as for the GCN-based model. Same as for GCN FORMULA9 ), we optimize the parameters of the MLP, DISPLAYFORM1 Free variable (FV): As an even more simple baseline, we consider treating the community affiliations F as a free variable in optimization. DISPLAYFORM2 This is similar to standard community detection methods like BigCLAM. Since this optimization problem is rather different from those of GCN FORMULA9 ) and MLP (Equation 8), we perform additional hyperparameter optimization for the FV model. We consider different choices for the learning rate and two initialization strategies, while keeping other aspects of the training procedure as before (stochastic training, early stopping). We pick the configuration that achieved the best average NMI score across all datasets. Note that this gives a strong advantage to the FV model, since for GCN and MLP models the hyperparameters were fixed without the knowledge of the ground-truth communities. Experimental setup. We compare the NMI scores obtained by all three models, both for BernoulliPoisson and sigmoid link functions. Results. As shown in TAB2, GNN-based models outperforms the simpler baselines in 16 out of 20 cases (Remember, that the free variable version even had the advantage of picking the hyperparmeters that lead to the highest NMI scores). This highlights the fact that attribute information only is not enough for community detection, and incorporating the graph structure clearly helps to make better inferences. So far, we have observed that the DOC model is able to recover communities with high precision. What's even more interesting, since our model learns the mapping from node attributes to the producing community affiliations (Equation 3), it should also be possible to predict communities inductively for nodes that were not present at training time. Experimental setup. We hide a randomly selected fraction of nodes from each community, and train the DOC-BP and DOC-Sigmoid models on the remaining graph. Once the parameters θ are learned, we perform a forward pass of each model using the full adjacency and attribute matrix. We then compute how well the communities were predicted for the nodes that were not present during training, using NMI as a metric. We compare with the MLP model (Equation 7) as a baseline. Results. As can be seen in FIG1, both DOC-BP and DOC-Sigmoid are able to infer communities inductively for previously unseen nodes with high accuracy (NMI ≥ 40%), which is on the same level as for the transductive setting TAB2. On the other hand, MLP-BP and MLP-Sigmoid models both perform worse than the GCN-based ones, and significantly below their own scores for transductive community detection. This highlights the fact that graph-based neural network architectures provide a significant advantage for community detection. The problem of community detection in graphs is well-established in the research community, and methods such as stochastic block models BID0 and spectral methods BID30 have attracted a lot of attention. Despite the popularity of these methods, they are only suitable for detecting non-overlapping communities (i.e. partitioning the network), which is not the setting usually encountered in real-world networks BID34. Methods for overlapping community detection have been proposed BID32, but our understanding of their behavior is not as mature as for the non-overlapping methods. As discussed in Section 2, methods for OCD can be broadly divided into methods based on nonnegative matrix factorization, probabilistic inference and heuristics. These categories are not mutually exclusive, and often one method can be viewed as belonging to multiple categories. For example, the factorization-based approaches that minimize the Frobenius norm A − F F DISPLAYFORM0 More generally, most NMF and probabilistic inference models are performing a non-linear low rank decomposition of the adjacency matrix, which can be connected to the generalized principle component analysis model BID5.Deep learning for graph data can be broadly subdivided into two main categories: graph neural networks and node embeddings. Graph neural networks BID14 ) are specialized neural network architectures that can operate on graph-structured data. The goal of embedding approaches BID23; BID1 is to learn vector representations of nodes in a graph, that can later be used for other downstream machine learning tasks. One can perform k-means clustering on the node embeddings (as done in, e.g., BID29) to cluster nodes into communities. However, such approach is not able to capture the overlapping community structure present in real-world graphs. Several works have devised deep learning methods for community detection in graphs. BID36 and BID3 propose deep learning approaches that seek a low-rank decomposition of the modularity matrix BID21. This means both of these approaches are limited to finding disjoint communities, as opposed to our algorithm. Also related to our model is the approach by BID12, where they use a deep belief network to generate the community affiliation matrix. However, their neural network architecture does not use the graph, which we have shown to be crucial in Section 4.4. Lastly, BID4 designed a neural network architecture for supervised community detection. Their model learns to detect communities by training on a labeled set with community information given. This is very different from this paper, where we learn to detect communities in a fully unsupervised manner. In this work we have proposed and studied two deep models for overlapping community detection: DOC-BP, based on the Bernoulli-Poisson link, and DOC-Sigmoid, that relies on the sigmoid link function. The two variants of our model achieve state-of-the-art and convincingly outperfom existing techniques in transductive and inductive community detection tasks. Using stochastic training, both approaches are highly efficient and scale to large graphs. Among the two proposed models, DOC-BP one average performs better than the DOC-Sigmoid variant. We leave to future work to investigate the properties of communities detected by these two methods. To summarize, the obtained in our experimental evaluation provide strong evidence that deep learning for graphs deserves more attention as a framework for overlapping community detection. Architecture. We use a 3-layer graph convolutional neural network (Equation 3), with hidden sizes of 64, and the final (third) layer has size C (number of communities to be detected). Dropout with 50% keep probability is applied at every layer. We don't use any other forms of regularization, such as weight decay. Training. We train the model using Adam optimizer with default parameters. The learning rate is set to 10 −4. We use the following early stopping strategy: Before training, we set aside 1% of edges and the same number of non-edges. Every 25 gradient steps we compute the loss (Equation 5) for the validation edges and non-edges. We stop training if there was no improvement to the best validation loss for 20 × 25 = 500 iterations, or after 5000 epochs, whichever happens first. Raw Amazon data is provided by BID19 at http://jmcauley.ucsd.edu/ data/amazon/links.html.• Nodes: A node in the graph represents a product sold at Amazon. To get a graph of manageable size, we restrict our attention to the products in 14 randomly-chosen subcategories of the "Video Games" supercategory.• Edges: A pair of products (u, v) is connected by an edge if u is "also bought" with product v, or the other way around. The "also bought" information is provided in the raw data.• Communities: We treat the subcategories (e.g. "Xbox 360", "PlayStation 4") as community labels. Every product can belong to multiple categories.• Features: The authors of BID11 extracted visual features from product pictures using a deep CNN. We use these visual features as attributes X in our experiments. We use the dump of the Microsoft Academic Graph that was published for the KDD CUP 2016 competition (https://kddcup2016.azurewebsites.net/) to construct two co-authorship graphs -Coauthor-CS and Coauthor-Physics.• Nodes: A node in the graph represents a researcher.• Edges: A pair of researchers (u, v) is connected by an edge if u and v have co-authored one or more papers.• Communities: For Computer Science (Coauthor-CS), we use venues as proxies for fields of study. We pick top-5 venues for each subfield of CS according to Google Scholar (scholar.google.com). An author u is assigned to field of study c if he published at least 5 papers in venues associated with this field of study. For Physics (Coauthor-Physics), we use the Physcial Review A, B, C, D, E journals as indicators of fields of study (= communities). An author u is assigned to field of study c if he published at least 5 papers in the respective Physical Review "?" journal.• Features: For author user u we construct a histogram over keywords that were assigned to their papers. That is, the entry of the attribute matrix X ud = # of papers that author u has published that have keyword d. For this graph we had to remove from our consideration the papers that had too many (≥ 40) authors, since it led to very large fully-connected components in the ing graph. Reddit is an online content-sharing platform, where users share, rate, and comment on content on a wide range of topics. The site consists of a number of smaller topic-oriented communi-ties, called subreddits. We downloaded a dump of Reddit comments for February 2018 from http://files.pushshift.io/reddit/comments/. Using the list provided at https: //www.reddit.com/r/ListOfSubreddits/, we picked 48 gaming-related subreddits and 31 technology-based subreddits. For each of these groups of subreddits we constructed a graph as following:• Nodes: A node in the graph represents a user of Reddit, identified by their author id.• Edges: A pair of users (u, v) is connected by an edge if u and v have both commented on the same 3 or more posts.• Communities: We treat subreddits as communities. A user is assigned to a community c, if he commented on at least 5 posts posted in that community.• Features: For every user u we construct a histogram of other subreddits (excluding the subreddits used as communities) that they commented in. That is, the entry of the attribute matrix X ud = # of comments user u left on subreddit d. Co-purchase graphs, co-authorship graphs and content-sharing platforms are classic examples of networks with overlapping community structure BID34, so using these communities as ground truth is justifiable. Additionally, we show that for all the five graphs considered, the probability of connection between a pair of nodes grows monotonically with the number of shared communities. This further shows that our choice of communities makes sense. • We used the reference C++ implementations of BigCLAM and CESNA, that were provided by the authors (https://github.com/snap-stanford/snap). Models were used with the default parameter settings for step size, backtracking line search constants, and balancing terms. Since CESNA can only handle binary attributes, we binarize the original attributes (set the nonzero entries to 1) if they have a different type.• We implemented SNMF ourselves using Python. The F matrix is initialized by sampling from the Uniform distribution. We run optimization until the improvement in the reconstruction loss goes below 10 −4 per iteration, or for 300 epochs, whichever happens first.• We use the Matlab implementation of CDE provided by the authors. We set the hyperparameters to α = 1, β = 2, κ = 5, as recommended in the paper, and run optimization for 20 iterations.• We use the Python implementation of LDense provided by the authors (https:// research.cs.aalto.fi/dmg/software.shtml), and run the algorithm with the recommended parameter settings. Same as with CESNA, since the methods only supports binary attributes, we binarize the original data if necessary.

Represent the short summary for finding the corresponding scientific article (long)

Detecting overlapping communities in graphs using graph neural networks

Represent this text to find a much much longer article about it with say hundreds-thousands of words

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Neural Architecture Search (NAS) aims to facilitate the design of deep networks for new tasks. Existing techniques rely on two stages: searching over the architecture space and validating the best architecture. NAS algorithms are currently compared solely based on their on the downstream task. While intuitive, this fails to explicitly evaluate the effectiveness of their search strategies. In this paper, we propose to evaluate the NAS search phase. To this end, we compare the quality of the solutions obtained by NAS search policies with that of random architecture selection. We find that: (i) On average, the state-of-the-art NAS algorithms perform similarly to the random policy; (ii) the widely-used weight sharing strategy degrades the ranking of the NAS candidates to the point of not reflecting their true performance, thus reducing the effectiveness of the search process. We believe that our evaluation framework will be key to designing NAS strategies that consistently discover architectures superior to random ones.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We empirically disprove a fundamental hypothesis of the widely-adopted weight sharing strategy in neural architecture search and explain why the state-of-the-arts NAS algorithms performs similarly to random search.

Represent the TLDR for finding the corresponding scientific article

We use reinforcement learning for query reformulation on two tasks and surprisingly find that when training multiple agents diversity of the reformulations is more important than specialisation.


Question: Represent the scientific article to find a short TLDR

Question: In this paper we use the geometric properties of the optimal transport (OT) problem and the Wasserstein distances to define a prior distribution for the latent space of an auto-encoder. We introduce Sliced-Wasserstein Auto-Encoders (SWAE), that enable one to shape the distribution of the latent space into any samplable probability distribution without the need for training an adversarial network or having a likelihood function specified. In short, we regularize the auto-encoder loss with the sliced-Wasserstein distance between the distribution of the encoded training samples and a samplable prior distribution. We show that the proposed formulation has an efficient numerical solution that provides similar capabilities to Wasserstein Auto-Encoders (WAE) and Variational Auto-Encoders (VAE), while benefiting from an embarrassingly simple implementation. We provide extensive error analysis for our algorithm, and show its merits on three benchmark datasets. Learning such generative models boils down to minimizing a dissimilarity measure between the data distribution and the output distribution of the generative model. To this end, and following the work of and, we approach the problem of generative modeling from the optimal transport point of view. The optimal transport problem BID38; BID22 provides a way to measure the distances between probability distributions by transporting (i.e., morphing) one distribution into another. Moreover, and as opposed to the common information theoretic dissimilarity measures (e.g., f -divergences), the p-Wasserstein dissimilarity measures that arise from the optimal transport problem: 1) are true distances, and 2) metrize a weak convergence of probability measures (at least on compact spaces). Wasserstein distances have recently attracted a lot of interest in the learning community BID11; BID14;;; BID22 due to their exquisite geometric characteristics BID34. See the supplementary material for an intuitive example showing the benefit of the Wasserstein distance over commonly used f -divergences. In this paper, we introduce a new type of auto-encoders for generative modeling (Algorithm 1), which we call Sliced-Wasserstein auto-encoders (SWAE), that minimize the sliced-Wasserstein distance between the distribution of the encoded samples and a samplable prior distribution. Our work is most closely related to the recent work by and more specifically the follow-up work by. However, our approach avoids the need to perform adversarial training in the encoding space and is not restricted to closed-form distributions, while still benefiting from a Wasserstein-like distance measure in the latent space. Calculating the Wasserstein distance can be computationally expensive, but our approach permits a simple numerical solution to the problem. Finally, we note that there has been several concurrent papers, including the work by BID10 Deshpande et al. andŞimşekli et al. (2018, that also looked into the application of sliced-Wasserstein distance in generative modeling. Regardless of the concurrent nature of these papers, our work remains novel and is distinguished from these methods. BID10 use the sliced-Wasserstein distance to match the distributions of high-dimensional reconstructed images, which require large number of slices, O(10 4), while in our method and due to the distribution matching in the latent space we only need O slices. We also note that BID10 proposed to learn discriminative slices to mitigate the need for a very large number of random projections that is in essence similar to the adversarial training used in GANs, which contradicts with our goal of not using adversarial training. Simşekli et al., on the other hand, take an interesting but different approach of parameter-free generative modeling via sliced-Wasserstein flows. Let X denote the compact domain of a manifold in Euclidean space and let x n ∈ X denote an individual input data point. Furthermore, let ρ X be a Borel probability measure defined on X. We define the probability density function p X (x) for input data x to be: DISPLAYFORM0 Let φ: X → Z denote a deterministic parametric mapping from the input space to a latent space Z (e.g., a neural network encoder). To obtain the density of the push forward of ρ X with respect to φ, i.e., ρ Z = φ * (ρ X), we use Random Variable Transformation (RVT) BID12 ). In short, the probability density function of the encoded samples z can be expressed in terms of φ and p X by: DISPLAYFORM1 where δ denotes the Dirac distribution function. Similar to variational Auto-Encoders (VAEs) BID18 and the Wasserstein Auto-Encoders (WAE), our main objective is to encode the input data points x ∈ X into latent codes z ∈ Z such that: 1) x can be recovered/approximated from z, and 2) the probability density function of the encoded samples, p Z, follows a prior distribution q Z. Let ψ: Z → X be the decoder that maps the latent codes back to the original space such that DISPLAYFORM2 where y denotes the decoded samples. It is straightforward to see that when ψ = φ −1 (i.e. ψ(φ(·)) = id(·)), the distribution of the decoder p Y and the input distribution p X are identical. Hence, in its most general form, the objective of such auto-encoders simplifies to learning φ and ψ, so that they minimize a dissimilarity measure between p Y and p X, and between p Z and q Z. In what follows, we briefly review the existing dissimilarity measures for these distributions.1.1 MINIMIZING DISSIMILARITY BETWEEN p X AND p Y We first emphasize that the VAE often assumes stochastic encoders and decoders BID18, while we consider the case of only deterministic mappings. Although, we note that, similar to WAE, SWAE can also be formulated with stochastic encoders. Different measures have been used previously to compute the dissimilarity between p X and p Y. Most notably, BID29 showed that for the general family of f -divergences, D f (p X, p Y), (including the KL-divergence, JensenShannon, etc.), using the Fenchel conjugate of the convex function f and minimizing D f (p X, p Y) leads to a min-max problem that is equivalent to the adversarial training widely used in the generative modeling literature BID13; BID26; BID27.Others have utilized the rich mathematical foundation of the OT problem and Wasserstein distances; BID14;; to define a distance between p X and p Y. In Wasserstein-GAN, utilized the Kantorovich-Rubinstein duality for the 1-Wasserstein distance, W 1 (p X, p Y), and reformulated the problem as a min-max optimization that is solved through an adversarial training scheme. Inspired by the work of and, it can be shown that (see supplementary material for a proof): DISPLAYFORM3 Furthermore, the r.h.s. of equation 3 supports a simple implementation where for i.i.d samples of the input distribution, {x n} N n=1, the upper bound can be approximated as: DISPLAYFORM4 The r.h.s of equation 3 and equation 4 take advantage of the existence of pairs x n and y n = ψ(φ(x n)), which make f (·) = ψ(φ(·)) a transport map between p X and p Y (but not necessarily the optimal transport map). In this paper, we minimize W ‡ c (p X, p Y) following equation 4 to minimize the discrepancy between p X and p Y. Next, we focus on the discrepancy measures between p Z and q Z.1.2 MINIMIZING DISSIMILARITY BETWEEN p Z AND q Z If q Z is a known distribution with an explicit formulation (e.g. Normal distribution) the most straightforward approach for measuring the (dis)similarity between p Z and q Z is the log-likelihood of z = φ(x) with respect to q Z, formally: DISPLAYFORM5 maximizing the log-likelihood is equivalent to minimizing the KL-divergence between p Z and q Z, D KL (p Z, q Z) (see supplementary material for more details and derivation of Equation equation 5). This approach has two major limitations: 1) The KL-Divergence and in general f -divergences do not provide meaningful dissimilarity measures for distributions supported on non-overlapping lowdimensional manifolds; (see supplementary material), which is common in hidden layers of neural networks, and therefore they do not provide informative gradients for training φ, and 2) we are limited to distributions q Z that have known explicit formulations, which is restrictive as it eliminates the ability to use the much broader class of samplable distributions. Various alternatives exist in the literature to address the above-mentioned limitations. These methods often sampleZ = {z j} N j=1 from q Z and Z = {z n = φ(x n)} N n=1 from p X and measure the discrepancy between these sets (i.e. point clouds). Note that there are no one-to-one correspondences betweenz j s and z n s. In their influential WAE paper, proposed two different approaches for measuring the discrepancy betweenZ and Z, namely the GAN-based and the maximum mean discrepancy (MMD)-based approaches. The GAN-based approach proposed in defines a discriminator network, D Z (p Z, q Z), to classifyz j s and z n s as coming from'true' and'fake' distributions correspondingly, and proposes a min-max adversarial optimization for learning φ and D Z. The MMD-based approach, utilizes a positive-definite reproducing kernel k: Z × Z → R to measure the discrepancy betweenZ and Z. The choice of the kernel and its parameterization, however, remain a data-dependent design parameter. An interesting alternative approach is to use the Wasserstein distance between p Z and q Z. Following the work of, this can be accomplished utilizing the Kantorovich-Rubinstein duality and through introducing a min-max problem, which leads to yet another adversarial training scheme similar to the GAN-based method in. Note that, since elements ofZ and Z are not paired, an approach similar to equation 4 could not be used to minimize the discrepancy. In this paper, we propose to use the sliced-Wasserstein metric, BID3; BID21; BID6;, to measure the discrepancy between p Z and q Z. We show that using the sliced-Wasserstein distance ameliorates the need for training an adversary network or choosing a data-dependent kernel (as in WAE-MMD), and provides an efficient, stable, and simple numerical implementation. Before explaining our proposed approach, it is worthwhile to point out the major difference between learning auto-encoders as generative models and GANs. In GANs, one needs to minimize a distance between {ψ(z j)|z j ∼ q Z } M j=1 and {x n} M n=1, which are high-dimensional point clouds for which there are no correspondences between ψ(z j)s and x n s. For the auto-encoders, on the other hand, there exists correspondences between the high-dimensional point clouds {x n} M n=1 and {y n = ψ(φ(x n))} M n=1, and the problem simplifies to matching the lower-dimensional point clouds {φ( DISPLAYFORM6 . In other words, the encoder performs a nonlinear dimensionality reduction, that enables us to solve a simpler problem compared to GANs. Next we introduce the details of our approach. In what follows we first provide a brief review of the necessary equations to understand the Wasserstein and sliced-Wasserstein distances and then present our Sliced Wasserstein auto-encoder (SWAE). The Wasserstein distance between probability measures ρ X and ρ Y, with corresponding densities dρ X = p X (x)dx and dρ Y = p Y (y)dy is defined as: DISPLAYFORM0 where DISPLAYFORM1 is the set of all transportation plans (i.e. joint measures) with marginal densities p X and p Y, and c: X × Y → R + is the transportation cost. equation 6 is known as the Kantorovich formulation of the optimal mass transportation problem, which seeks the optimal transportation plan between p X and p Y. If there exist diffeomorphic mappings, f: X → Y (i.e. transport maps) such that y = f (x) and consequently, DISPLAYFORM2 where det(D·) is the determinant of the Jacobian, then the Wasserstein distance could be defined based on the Monge formulation of the problem (see BID38 and BID22) as: DISPLAYFORM3 where M P is the set of all diffeomorphisms that satisfy equation 7. As can be seen from equation 6 and equation 8, obtaining the Wasserstein distance requires solving an optimization problem. We note that various efficient optimization techniques have been proposed in the past (e.g. Cuturi FORMULA1 ; BID36 ; Oberman & Ruan FORMULA1) to solve this optimization. For one-dimensional probability densities, p X and p Y, however, the Wasserstein distance has a closed-form solution. Let P X and P Y be the cumulative distributions of one-dimensional probability distributions p X and p Y, correspondingly. The Wassertein distance can then be calculated as below (see BID22 for more details): DISPLAYFORM4 This closed-form solution motivates the definition of sliced-Wasserstein distances. Sliced-Wasserstein distance has similar qualitative properties to the Wasserstein distance, but it is much easier to compute. The sliced-Wasserstein distance was used in to calculate barycenter of distributions and point clouds. BID3 provided a nice theoretical overview of barycenteric calculations using the sliced-Wasserstein distance. BID21 used it to define positive definite kernels for distributions BID6 to define a kernel for persistence diagrams. Sliced-Wasserstein was recently used for learning Gaussian mixture models in, and it was also used as a measure of goodness of fit for GANs in BID17.The main idea behind the sliced-Wasserstein distance is to slice (i.e., project) higher-dimensional probability densities into sets of one-dimensional marginal distributions and compare these marginal distributions via the Wasserstein distance. The slicing/projection process is related to the field of Integral Geometry and specifically the Radon transform (see BID15). The relevant to our discussion is that a d-dimensional probability density p X can be uniquely represented as the set of its one-dimensional marginal distributions following the Radon transform and the Fourier slice theorem BID15. These one dimensional marginal distributions of p X are defined as: DISPLAYFORM0 where S d−1 is the d-dimensional unit sphere. Note that for any fixed θ ∈ S d−1, Rp X (·; θ) is a one-dimensional slice of distribution p X. In other words, Rp X (·; θ) is a marginal distribution of p X that is obtained from integrating p X over the hyperplane orthogonal to θ. Utilizing these marginal distributions in equation 10, the sliced Wasserstein distance could be defined as: DISPLAYFORM1 Given that Rp X (·; θ) and Rp Y (·; θ) are one-dimensional, the Wasserstein distance in the integrand has a closed-form solution (see equation 9). Moreover, it can be shown that SW c is a true metric BID4 and BID20 ), and it induces the same topology as W c, at least on compact sets BID34. A natural transportation cost that has extensively studied in the past is the 2 2, c(x, y) = x − y 2 2, for which there are theoretical guarantees on existence and uniqueness of transportation plans and maps (see BID34 and BID38). When c(x, y) = x − y p p for p ≥ 2, the following upper bound hold for the SW distance: DISPLAYFORM2 where, DISPLAYFORM3 Chapter 5 in BID4 proves this inequality. In our paper, we are interested in p = 2, for which α p,d = 1 d, and we have: DISPLAYFORM4 In the Numerical Implementation Section, we provide a numerical experiment to compare W 2 and SW 2, that confirms the above equation. Our proposed formulation for the SWAE is as follows: DISPLAYFORM0 where φ is the encoder, ψ is the decoder, p X is the data distribution, p Y is the data distribution after encoding and decoding (equation 2), p Z is the distribution of the encoded data (equation 1), q Z is a predefined samplable distribution, and λ indicates the relative importance of the loss functions. To further clarify why we use the sliced-Wasserstein distance to measure the difference between p Z and q Z, we reiterate that due to the lack of correspondences betweenz i s and z j s, one cannot minimize the upper-bound in equation 4, and calculation of the Wasserstein distance requires an additional optimization step to obtain the optimal coupling between p Z and q Z. To avoid this additional optimization, while maintaining the favorable characteristics of the Wasserstein distance, we use the sliced-Wasserstein distance to measure the discrepancy between p Z and q Z. We now describe the numerical details of our approach. The Wasserstein distance between two one-dimensional probability densities p X and p Y is obtained from equation 9. The integral in equation 9 can be numerically estimated using the midpoint Riemann sum, DISPLAYFORM0 2M (see FIG0). In scenarios where only samples from the distributions are available, x m ∼ p X and y m ∼ p Y, the empirical densities can be estimated as DISPLAYFORM1 δ ym, where δ xm is the Dirac delta function centered at x m. Therefore the corresponding empirical distribution function of p X is P X (t) ≈ P X,M (t) = 1 M M m=1 u(t−x m) where u is the step function (P Y,M (t) is defined similarly). From Glivenko-Cantelli Theorem we have that sup t |P X,M (t) − P X (t)| a.s.− − → 0, where the convergence behavior is achieved via Dvoretzky-Kiefer-Wolfowitz inequality bound: DISPLAYFORM2 2 ). Calculating the Wasserstein distance with the empirical distribution function is computationally attractive. Sorting x m s in an ascending order, such that DISPLAYFORM3 and where i[m] is the index of the sorted x m s, it is straightforward to see that P −1 FIG0. The Wasserstein distance can be approximated by first sorting x m s and y m s and then calculating: We need to address one final question here. How well does equation 15 approximate the Wasserstein distance, W c (p X, p Y)? We first note that the rates of convergence of empirical distributions, for the p-Wasserstein metric (i.e., c(x, y) = |x − y| p ) of order p ≥ 1, have been extensively studied in the mathematics and statistics communities (see for instance BID2 and BID9). A detailed description of these rates is, however, beyond the scope of this paper, especially since these rates are dependent on the choice of p. In short, for p = 1 it can be shown that DISPLAYFORM4 DISPLAYFORM5 DISPLAYFORM6 where C is an absolute constant. Similar are achieved for E(Wp(pX,M, p X)) and (E(W p p (p X,M, p X))) 1 p, although under more strict assumptions on p X (i.e., slightly stronger assumptions than having a finite second moment). Using the triangle inequality together with the convergence rates of empirical distributions with respect to the p-Wasserstein distance, see BID2, for W 1 (p X,M, p X) (or more generally W p (p X,M, p X)) we can show that (see supplementary material): DISPLAYFORM7 for some absolute constant, C. We reiterate that similar bounds could be found for W p although with slightly more strict assumptions on p X and p Y. In scenarios where only samples from the d-dimensional distribution, p X, are available, x m ∼ p X, the empirical density can be estimated as p X,M = 1 M M m=1 δ xm. Following equation 10 it is straightforward to show that the marginal densities (i.e. slices) are obtained from: DISPLAYFORM0 see the supplementary material for a proof. The Dvoretzky-Kiefer-Wolfowitz upper bound holds for Rp X (t, θ) and Rp X,M (t, θ). Minimizing the sliced-Wasserstein distance (i.e., as in the second term of 14) requires an integration over the unit sphere in R d, i.e., S d−1. In practice, this integration is approximated by using a simple Monte Carlo scheme that draws uniform samples from S d−1 and replaces the integral with a finite-sample average, DISPLAYFORM0 Figure 2: SW approximations (scaled by 1.22 DISPLAYFORM1, and different number of random slices, L. Moreover, the global minimum for SW c (p Z, q Z) is also a global minimum for each DISPLAYFORM2 A fine sampling of S d−1, however, is required for a good approximation of SW c (p Z, q Z). Intuitively, if p Z and q Z are similar, then their projections with respect to any finite subset of S d−1 would also be similar. This leads to a stochastic gradient descent scheme where in addition to the random sampling of the input data, we also random sample the projection angles from S d−1.A natural question arises on the effect of the number of random slices, L = |Θ|, on the approximation of the SW distance. Here, we devised a simple experiment that demonstrates the effect of L on approximating the SW distance. We generated two random multi-variate Gaussian distributions in a d-dimensional space, where d ∈ {2 n} 10 n=1, to serve as p X = N (µ X, Σ X) and p X = N (µ Y, Σ Y). The Wasserstein distance for the two Gaussian distributions has a closed form solution, DISPLAYFORM3 which served as the ground-truth distance between the distributions. We then measured the SW distance between M = 1000 samples generated from the two Gaussian distributions using L ∈ {1, 10, 50, 100, 500, 1000} random slices. We repeated the experiment for each L and d, a thousand times and report the means and standard deviations in Figure 2. Following equation 13 we scaled the SW distance by √ d. Moreover we found out empirically that 1.22 DISPLAYFORM4 It can be seen from Figure 2 that the expected value of the scaled SW -distance closely follows the true Wasserstein distance. A more interesting observation is that the variance of estimation increases for higher dimensions d and decreases as the number of random projections, L, increases. Hence, calculating the SW distance in the image space, as in BID10, requires a very large number of projections L to get a less variant approximation of the distance. be randomly sampled from a uniform distribution on S d−1. Then using the numerical approximations described in this section, the loss function in equation 14 can be rewritten as: The steps of our proposed method are presented in Algorithm 1. It is worth pointing out that sorting is by itself an optimization problem (which can be solved very efficiently), and therefore the sorting followed by the gradient descent update on φ and ψ is in essence a min-max problem, which is being solved in an alternating fashion. Finally, we point out that each iteration of SWAE costs O(LM log(M)) operations. DISPLAYFORM5 Require: Regularization coefficient λ, and number of random projections, L. Initialize the parameters of the encoder, φ, and decoder, ψ while φ and ψ have not converged do Sample {x1, ..., xM} from training set (i.e. pX) Sample {z1, ...,zM} from qZ Sample {θ1, ..., θL} from S DISPLAYFORM0 Update φ and ψ by descending: DISPLAYFORM1 In our experiments we used three image datasets, namely the MNIST dataset by BID24, the CelebFaces Attributes Dataset (CelebA) by BID25, and the LSUN Bedroom Dataset by BID41. For the MNIST dataset we used a simple auto-encoder with mirrored classic deep convolutional neural networks with 2D average poolings, leaky rectified linear units (Leaky-ReLu) as the activation functions, and upsampling layers in the decoder. For the CelebA and LSUN datasets we used the architecture similar to.To test the capability of our proposed algorithm in shaping the latent space of the encoder, we started with the MNIST dataset and trained SWAE to encode this dataset to a two-dimensional latent space (for the sake of visualization) while enforcing a match between p X and p Y and p Z and q Z. We chose four different samplable distributions as shown in FIG5. It can bee seen that SWAE can successfully embed the dataset into the latent space while enforcing p Z to closely follow q Z. In addition, we sample the two-dimensional latent spaces on a 25 × 25 grid in [−1, 1] 2 and decode these points to visualize their corresponding images in the digit/image space. To get a sense of the convergence behavior of SWAE, and similar to the work of BID17, we calculate the Sliced Wasserstein distance between p Z and q Z as well as p X and p Y at each batch iteration where we used p-LDA BID39 to calculate projections (See supplementary material). We compared the convergence behavior of SWAE with the closest related work, (specifically WAE-GAN) where an adversarial training is used to match p Z to q Z, while the loss function for p X and p Y remains exactly the same between the two methods. We repeated the experiments 100 times and report the summary of in FIG2. We mention that the exact same models and optimizers were used for both methods in this experiment. An interesting observation, here is that while WAE-GAN provides good or even slightly better generated random samples for MNIST (lower sliced-Wasserstein distance between p X and p Y), it fails to provide a good match between p Z and q Z for the choice of the prior distribution reported in FIG2. This phenomenon seems to be related to the mode-collapse problem of GANs, where the adversary fails to sense that the distribution is not fully covered. Finally, in our experiments we did not notice a significant difference between the computational time for SWAE and WAE-GAN. For the MNIST experiment and on a single NVIDIA Tesla P 100 GPU, each batch iteration (batchsize=500) of WAE-GAN took 0.2571 ± 0.0435(sec) while SWAE (with L = 50 projections) took 0.2437 ± 0.0391(sec). DISPLAYFORM0 The distribution in the 64-dimensional latent space, q Z, was set to Normal. We also report the negative log-likelihood of {z i = φ(x i)} with repect to q Z for 1000 testing samples for both datasets. We did not use Nowizin's trick for the GAN models. DISPLAYFORM1 True Data 2 3 Table 2: FID score statistics (N = 5) at final iteration of training. Lower is better. Scores were computed with 10 4 random samples from the testing set against an equivalent amount of generated samples. The CelebA face and the LSUN bedroom datasets contain higher degrees of variations compared to the MNIST dataset and therefore a two-dimensional latent-space does not suffice to capture the variations in these datasets (See supplementary material for more details on the dimensionality of the latent space). We used a K = 64 dimensional latent spaces for both the CelebA and the LSUN Bedroom datasets, and also used a larger auto-encoder (i.e., DCGAN, following the work of). For these datasets SWAE was trained with q Z being the Normal distribution to enable the calculation of the negative log likelihood (NLL). TAB1 shows the comparison between SWAE and WAE for these two datasets. We note that all experimental parameters were kept the same to enable an apples to apples comparison. Finally, FIG4 demonstrates the interpolation between two sample points in the latent space, i.e. ψ(tφ(I 0) + (1 − t)φ(I 1)) for t ∈, for all three datasets. We introduced Sliced Wasserstein auto-encoders (SWAE), which enable one to shape the distribution of the encoded samples to any samplable distribution without the need for adversarial training or having a likelihood function specified. In addition, we provided a simple and efficient numerical scheme for this problem, which only relies on few inner products and sorting operations in each SGD iteration. We further demonstrated the capability of our method on three image datasets, namely the MNIST, the CelebA face, and the LSUN Bedroom datasets, and showed competitive performance, in the sense of matching distributions p Z and q Z, to the techniques that rely on additional adversarial trainings. Finally, we envision SWAE could be effectively used in transfer learning and domain adaptation algorithms where q Z comes from a source domain and the task is to encode the target domain p X in a latent space such that the distribution follows the distribution of the target domain. Figure 4: Sample convergence behavior for our method compared to the WAE-GAN, where q Z is set to a ring distribution FIG5, top left). The columns represent batch iterations (batchsize= 500).The top half of the table shows of ψ(z) for z ∼ q Z, and the bottom half shows z ∼ q Z and φ(x) for x ∼ p X. It can be seen that the adversarial loss in the latent space does not provide a full coverage of the distribution, which is a similar problem to the well-known'mode collapse' problem in the GANs. It can be seen that SWAE provides a superior match between p Z and q Z while it does not require adversarial training. Following the example by and later here we show a simple example comparing the Jensen-Shannon divergence with the Wasserstein distance. First note that the Jensen-Shannon divergence is defined as, DISPLAYFORM0 where DISPLAYFORM1 q(x) )dx is the Kullback-Leibler divergence. Now consider the following densities, p(x) be a uniform distribution around zero and let q τ (x) = p(x − τ) be a shifted version of the p. FIG6 show W 1 (p, q τ) and JS(p, q τ) as a function of τ. As can be seen the JS divergence fails to provide a useful gradient when the distributions are supported on non-overlapping domains. To maximize (minimize) the similarity (dissimilarity) between p Z and q Z, we can write: DISPLAYFORM0 where we replaced p Z with equation 1. Furthermore, it is straightforward to show: DISPLAYFORM1 The Wasserstein distance between the two probability measures ρ X and ρ Y with respective densities p X and p Y, can be measured via the Kantorovich formulation of the optimal mass transport problem: DISPLAYFORM2 c(x, y)γ(x, y)dxdy DISPLAYFORM3 } is the set of all transportation plans (i.e., couplings or joint distributions) over p X and p Y. Now, note that the two step process of encoding p X into the latent space Z and decoding it to p Y, provides a unique decomposition of γ as γ 0 (x, y) = δ(y − ψ(φ(x)))p X (x) ∈ Γ. The optimal coupling (i.e., transport plan) between p X and p Y could be equal or different from γ(x, y) = δ(y −ψ(φ(x)))p X (x). This leads to the scenario on the right where DISPLAYFORM4 Therefore we can write: DISPLAYFORM5 which proves equation 3. Finally, taking the infimum of the two sides of the inequality, with respect to φ and ψ, we have: DISPLAYFORM6 is non-zero. Finally, we note that ψ(φ(·)) = id(·) is a global optima for both W c (p X, p Y) and W ‡ c (p X, p Y). Following equation 10 a distribution can be sliced via: Figure 9 demonstrates the outputs of trained SWAEs with K = 2 and K = 128 for sample input images. The input images were resized to 64 × 64 and then fed to our auto-encoder structure. This effect can also be seen for the MNIST dataset as shown in FIG0. When the dimensionality of the latent-space (i.e. information bottleneck) is too low the latent space will not contain enough information to reconstruct crisp images. Increasing the dimensionality of the latent space leads to crisper images. DISPLAYFORM0 In this paper we also used the sliced Wasserstein distance as a measure of goodness of fit (for convergence analysis). To provide a fair comparison between different methods, we avoided random projections for this comparison. Instead, we calculated a discriminant subspace to separate ψ(z) from ψ(φ(x)) for z ∼ q Z and x ∼ p X, and set the projection parameters θs to the calculated discriminant components. This will lead to only slices that contain discriminant information. We point out that the linear discriminant analysis (LDA) is not a good choice for this task as it only leads to one discriminant component (because we only have two classes). We used the penalized linear discriminant analysis (p-LDA) that utilizes a combination of LDA and PCA. In short, p-LDA solves the following objective function:argmax θ θ T S T θ θ T (S W + αI)θ s.t. θ = 1 where S W is the within class covariance matrix, S T is the data covariance matrix, I is the identity matrix, and α identifies the interpolation between PCA and LDA (i.e. α = 0 leads to LDA and α → ∞ leads to PCA). For p ≥ 1 we can use the triangle inequality and write DISPLAYFORM0 which leads to DISPLAYFORM1 Taking the expectation of both sides of the inequality and using the empirical convergence bounds of W p (in this case W 1) we have, DISPLAYFORM2 for some absolute constant C, where the last line comes from the empirical convergence bounds of distributions with respect to the Wasserstein distance, see BID2.

represent the tldr for finding the corresponding scientific article

In this paper we use the sliced-Wasserstein distance to shape the latent distribution of an auto-encoder into any samplable prior distribution.

An empirical study of variational inference based on chi-square divergence minimization, showing that minimizing the CUBO is trickier than maximizing the ELBO