Question: Represent the article for finding a single-sentence TLDR

Question: The conventional approach to solving the recommendation problem greedily ranks individual document candidates by prediction scores. However, this method fails to optimize the slate as a whole, and hence, often struggles to capture biases caused by the page layout and document interdepedencies. The slate recommendation problem aims to directly find the optimally ordered subset of documents (i.e. slates) that best serve users’ interests. Solving this problem is hard due to the combinatorial explosion of document candidates and their display positions on the page. Therefore we propose a paradigm shift from the traditional viewpoint of solving a ranking problem to a direct slate generation framework. In this paper, we introduce List Conditional Variational Auto-Encoders (ListCVAE), which learn the joint distribution of documents on the slate conditioned on user responses, and directly generate full slates. Experiments on simulated and real-world data show that List-CVAE outperforms greedy ranking methods consistently on various scales of documents corpora. Recommender systems modeling is an important machine learning area in the IT industry, powering online advertisement, social networks and various content recommendation services BID0 ). In the context of document recommendation, its aim is to generate and display an ordered list of "documents" to users (called a "slate" in BID2 ; BID3), based on both user preferences and documents content. For large scale recommender systems, a common scalable approach at inference time is to first select a small subset of candidate documents S out of the entire document pool D. This step is called "candidate generation". Then a function approximator such as a neural network (e.g., a Multi-Layer Perceptron (MLP)) called the "ranking model" is used to predict probabilities of user engagements for each document in the small subset S and greedily generates a slate by sorting the top documents from S based on estimated prediction scores BID4. This two-step process is widely popular to solve large scale recommendation problems due to its scalability and fast inference at serving time. The candidate generation step can decrease the number of candidates from millions to hundreds or less, effectively dealing with scalability when faced with a large corpus of documents D. Since |S| is much smaller than |D|, the ranking model can be reasonably complicated without increasing latency. However, there are two main problems with this approach. First the candidate generation and the ranking models are not trained jointly, which can lead to having candidates in S that are not the highest scoring documents of the ranking model. Second and most importantly, the greedy ranking method suffers from numerous biases that come with the visual presentation of the slate and context in which documents are presented, both at training and serving time. For example, there exists positional biases caused by users paying more attention to prominent slate positions BID5, and contextual biases, due to interactions between documents presented together in the same slate, such as competition and complementarity, relative attractiveness, etc..In this paper, we propose a paradigm shift from the traditional viewpoint of solving a ranking problem to a direct slate generation framework. We consider a slate "optimal" when it maximizes some type of user engagement feedback, a typical desired scenario in recommender systems. For example, given a database of song tracks, the optimal slate can be an ordered list (in time or space) of k songs such that the user ideally likes every song in that list. Another example considers news articles, the optimal slate has k ordered articles such that every article is read by the user. In general, optimality can be defined as a desired user response vector on the slate and the proposed model should be agnostic to these problem-specific definitions. Solving the slate recommendation problem by direct slate generation differs from ranking in that first, the entire slate is used as a training example instead of single documents, preserving numerous biases encoded into the slate that might influence user responses. Secondly, it does not assume that more relevant documents should necessarily be put in earlier positions in the slate at serving time. Our model directly generates slates, taking into account all the relevant biases learned through training. In this paper, we apply Conditional Variational Auto-Encoders (CVAEs) BID7 BID8 to model the distributions of all documents in the same slate conditioned on the user response. All documents in a slate along with their positional, contextual biases are jointly encoded into the latent space, which is then sampled and combined with desired conditioning for direct slate generation, i.e. sampling from the learned conditional joint distribution. Therefore, the model first learns which slates give which type of responses and then directly generates similar slates given a desired response vector as the conditioning at inference time. We call our proposed model List-CVAE. The key contributions of our work are:1. To the best of our knowledge, this is the first model that provides a conditional generative modeling framework for slate recommendation by direct generation. It does not necessarily require a candidate generator at inference time and is flexible enough to work with any visual presentation of the slate as long as the ordering of display positions is fixed throughout training and inference times.2. To deal with the problem at scale, we introduce an architecture that uses pretrained document embeddings combined with a negatively downsampled k-head softmax layer within the List-CVAE model, where k is the slate size. The structure of this paper is the following. First we introduce related work using various CVAE-type models as well as other approaches to solve the slate generation problem. Next we introduce our List-CVAE modeling approach. The last part of the paper is devoted to experiments on both simulated and the real-world datasets.2 RELATED WORK Traditional matrix factorization techniques have been applied to recommender systems with success in modeling competitions such as the Netflix Prize BID10. Later research emerged on using autoencoders to improve on the of matrix factorization BID11 (CDAE, CDL). More recently several works use Boltzmann Machines BID13 and variants of VAE models in the Collaborative Filtering (CF) paradigm to model recommender systems BID14 BID15 BID16 ) (Collaborative VAE, JMVAE, CVAE-CF, JVAE-CF). See FIG0 for model structure comparisons. In this paper, unless specified otherwise, the user features and any context are routinely considered part of the conditioning variables (in Appendix A Personalization Test, we test List-CVAE generating personalized slates for different users). These models have primarily focused on modeling individual document or pairs of documents in the slate and applying greedy ordering at inference time. Our model is also using a VAE type structure and in particular, is closely related to the Joint Multimodel Variational Auto-Encoder (JMVAE) architecture FIG0 ). However, we use whole slates as input instead of single documents, and directly generate slates instead of using greedy ranking by prediction scores. Other relevant work from the Information Retrieval (IR) literature are listwise ranking methods BID17 BID18 BID19 BID20 BID21. These methods use listwise loss functions that take the contexts and positions of training examples into account. However, they eventually assign a prediction score for each document and greedily rank them at inference time. In the Reinforcement Learning (RL) literature, BID3 view the whole slates as actions and use a deterministic policy gradient update to learn a policy that generates these actions, given concatenated document features as input. Finally, the framework proposed by BID22 predicts user engagement for document and position pairs. It optimizes whole page layouts at inference time but may suffer from poor scalability due to the combinatorial explosion of all possible document position pairs. We formally define the slate recommendation problem as follows. Let D denote a corpus of documents and let k be the slate size. Then let r = (r 1, . . ., r k) be the user response vector, where r i ∈ R is the user's response on document d i. For example, if the problem is to maximize the number of clicks on a slate, then let r i ∈ {0, 1} denote whether the document d i is clicked, and thus an optimal slate DISPLAYFORM0 Variational Auto-Encoders (VAEs) are latent-variable models that define a joint density P θ (x, z) between observed variables x and latent variables z parametrized by a vector θ. Training such models requires marginalizing the latent variables in order to maximize the data likelihood P θ (x) = P θ (x, z)dz. Since we cannot solve this marginalization explicitly, we resort to a variational approximation. For this, a variational posterior density Q φ (z|x) parametrized by a vector φ is introduced and we optimize the variational Evidence Lower-Bound (ELBO) on the data loglikelihood: DISPLAYFORM0 DISPLAYFORM1 where KL is the Kullback-Leibler divergence and where P θ (z) is a prior distribution over latent variables. In a Conditional VAE (CVAE) we extend the distributions P θ (x, z) and Q φ (z|x) to also depend on an external condition c. The corresponding distributions are indicated by P θ (x, z|c) and Q φ (z|x, c). Taking the conditioning c into account, we can write the variational loss to minimize as DISPLAYFORM2 We assume that the slates s = (d 1, d 2, . . . d k) and the user response vectors r are jointly drawn from a distribution P D k ×R k. In this paper, we use a CVAE to model the joint distribution of all DISPLAYFORM0 DISPLAYFORM1 is the conditioning vector, where r = (r 1, r 2, . . ., r k) is the user responses on the slate s. The concatenation of s and c makes the input vector to the encoder. The latent variable z ∈ R m has a learned prior distribution N (µ 0, σ 0). The raw output from the decoder are k vectors x 1, x 2..., x k, each of which is mapped to a real document through taking the dot product with the matrix Φ containing all document embeddings. Thus produced k vectors of logits are then passed to the negatively downsampled k-head softmax operation. At inference time, c is the ideal condition whose concatenation with sampled z is the input to the decoder. documents in the slate conditioned on the user responses r, i.e. P(d 1, d 2, . . . d k |r). At inference time, the List-CVAE model attempts to generate an optimal slate by conditioning on the ideal user response r.As we explained in Section 1, "optimality" of a slate depends on the task. With that in mind, we define the mapping Φ: R k → C. It transforms a user response vector r into a vector in the conditioning space C that encodes the user engagement metric we wish to optimize for. For instance, if we want to maximize clicks on the slate, we can use the binary click response vectors and set the conditioning to c = Φ(r):= k i=0 r i. Then at inference time, the corresponding ideal user response r would be (1, 1, . . ., 1), and correspondingly the ideal conditioning would be c = Φ(r) = k i=0 1 = k. As usual with CVAEs, the decoder models a distribution P θ (s|z, c) that, conditioned on z, is easy to represent. In our case, P θ (s|z, c) models an independent probability for each document on the slate, represented by a softmax distribution. Note that the documents are only independent to each other conditional on z. In fact, the marginalized posterior P θ (s|c) = z P θ (s|z, c)P θ (z|c)dz can be arbitrarily complex. When the encoder encodes the input slate s into the latent space, it learns the joint distribution of the k documents in a fixed order, and thus also encodes any contextual, positional biases between documents and their respective positions into the latent variable z. The decoder learns these biases through reconstruction of the input slates from latent variables z with conditions. At inference time, the decoder reproduces the input slate distribution from the latent variable z with the ideal conditioning, taking into account all the biases learned during training time. Figure 3: Predictive prior distribution of the latent variable z in R 2, conditioned on ideal user response c = (1, 1, . . ., 1). The color map corresponds to the expected total responses of the corresponding slates. Plots are generated from the simulation experiment with 1000 documents and slate size 10.To shed light onto what is encoded in the latent space, we simplify the prior distribution of z to be a fixed Gaussian distribution N (0, I) in R 2. We train List-CVAE and plot the predictive prior z. As training evolves, generated output slates with low total responses are pushed towards the edge of the latent space while high response slates cluster towards a growing center area (Figure 3). Therefore after training, if we sample z from its prior distribution N (0, I) and generate the corresponding output slates, they are more likely to have high total responses. Since the number of documents in D can be large, we first embed the documents into a low dimensional space. Let Ψ: D → S q−1 be that normalized embedding where S q−1 denotes the unit sphere in R q. Ψ can easily be pretrained using a standard supervised model that predicts user responses from documents or through a standard auto-encoder technique. For the i-th document in the slate, our model produces a vector x i in R q that is then matched to each document from D via a dot-product. This operation produces k vectors of logits for k softmaxes, i.e. the k-head softmax. At training time, for large document corpora D, we uniformly randomly downsample negative documents and compute only a small subset of the logits for every training example, therefore efficiently scaling the nearest neighbor search to millions of documents with minimal model quality loss. We train this model as a CVAE by minimizing the sum of the reconstruction loss and the KLdivergence term: DISPLAYFORM2 where β is a function of the training step BID23.During inference, output slates are generated by first sampling z from the conditionally learned prior distribution N (µ, σ), concatenating with the ideal condition c = Φ(r), and passed into the decoder, generating (x 1, . . ., x k) from the learned P θ (s|z, c), and finally taking arg max over the dot-products with the full embedding matrix independently for each position i = 1,..., k. for i = 1,..., k, where B represents the Bernoulli distribution. During training, all models see uniformly randomly generated slates s ∼ U({1, n} k ) and their generated responses r. During inference time, we generate slates s by conditioning on c = (1, . . ., 1). We do not require document de-duplication since repetition may be desired in certain applications (e.g. generating temporal slates in an online advertisement session). Moreover List-CVAE should learn to produce the optimal slates whether those slates contain duplication or not from learning the training data distribution. Evaluation: For evaluation, we cannot use offline ranking evaluation metrics such as Normalized Discounted Cumulative Gain (NDCG) BID24, Mean Average Precision (MAP) or Inverse Propensity Score (IPS) BID26, etc. These metrics either require prediction scores for individual documents or assume that more relevant documents should appear in earlier ranking positions, unfairly favoring greedy ranking methods. Moreover, we find it limiting to use various diversity metrics since it is not always the case that a higher diversity-inclusive score gives better slates measured by user's total responses. Even though these metrics may be more transparent, they do not measure our end goal, which is to maximize the expected number of total responses on the generated slates. Instead, we evaluate the expected number of clicks over the distribution of generated slates and over the distribution of clicks on each document: DISPLAYFORM0 In practice, we distill the simulated environment of Eq. 5 using the cross-entropy loss onto a neural network model that officiates as our new simulation environment. The model consists of an embedding layer, which encodes documents into 8-dimensional embeddings. It then concatenates the embeddings of all the documents that form a slate and follows this concatenation with two hidden layers and a final softmax layer that predicts the slate response amongst the 2 k possible responses. Thus we call it the "response model". We use the response model to predict user responses on 100,000 sampled output slates for evaluation purposes. This allows us to accurately evaluate our output slates by List-CVAE and all other baseline models. Our experiments compare List-CVAE with several greedy ranking baselines that are often deployed in industry productions, namely Greedy MLP, Pairwise MLP, Position MLP and Greedy Long Short-Term Memory (LSTM) models. In addition to the greedy baselines, we also compare against auto-regressive (AR) versions of Position MLP and LSTM, as well as randomly-selected slates from the training set as a sanity check. List-CVAE generates slates s = arg max s∈{1,...,n} k P θ (s|z, c). The encoder and decoder of List-CVAE, as well as all the MLP-type models consist of two fully-connected neural network layers of the same size. Greedy MLP trains on (d i, r i) pairs and outputs the greedy slate consisting of the top k highestP (r|d) scoring documents. Pairwise MLP is an MLP model with a pairwise ranking loss DISPLAYFORM0 where L x is the cross entropy loss and (x 1, x 2) are pairs of documents randomly selected with different user responses from the same slate. We sweep on hyperparameters α and η in addition to the shared MLP model structure sweep. Position MLP uses position in the slate as a feature during training time and simply sets it to 0 for fast performance at inference time. AR Position MLP is identical to Position MLP with the exception that the position feature is set to each corresponding slate position at inference time (as such it takes into account position biases). Greedy LSTM is an LSTM model with fully-connect layers before and after the recurrent middle layers. We tune the hyperparameters corresponding to the number of layers and their respective widths. We use sequences of documents that form slates as the input at training time, and use single examples as inputs with sequence length 1 at inference time, which is similar to scoring documents as if they are in the first position of a slate of size 1. Then we greedily rank the documents based on their prediction scores. AR LSTM is identical to Greedy LSTM during training. During inference, however, it selects documents sequentially by first selecting the best document for position 1, then unrolling the LSTM for 2 steps to select the best document for position 2, and so on. This way it takes into account the context of all previous documents and their positions. Random selects slates uniformly randomly from the training set. Small-scale experiment (n = 100, 1000, k = 10): 0 1000 2000 3000 4000 5000Step FORMULA1 Step FORMULA2 We use the trained document embeddings from the response model for List-CVAE and all the baseline models. For List-CVAE, we also use trained priors P θ (z |c) = N (µ, σ) where µ, σ = f prior (c) and f prior is modeled by a small MLP. Additionally, since we found little difference between different hyperparameters, we fixed the width of all hidden layers to 128, the learning rate to 10 DISPLAYFORM1 and the number of latent dimensions to 16. For all other baseline models, we sweep on the learning rates, model structures and any model specific hyperparameters such as α, η for Position MLP and the forget bias for the LSTM model. FIG5 show the performance comparison when the number of documents n = 100, 1000 and slate size to k = 10. While List-CVAE is not quite capable of reaching a perfect performance of 10 clicks (which is probably even above the optimal upper bound), it easily outperforms all other ranking baselines after only a few training steps. Appendix A includes an additional personalization test. Due to a lack of publicly available large scale slate datasets, we use the data provided by the RecSys 2015 YOOCHOOSE Challenge BID27. This dataset consists of 9.2M user purchase sessions around 53K products. Each user session contains an ordered list of products on which the user clicked, and whether they decided to buy them. The List-CVAE model can be used on slates with temporal ordering. Thus we form slates of size 5 by taking consecutive clicked products. We then build user responses from whether the user bought them. We remove a portion of slates with no positive responses such that after removal they only account for 50% of the total number of slates. After filtering out products that are rarely bought, we get 375K slates of size 5 and a corpus of 10,000 candidate documents. FIG9 shows the user response distribution of the training data. Notice that in the response vector, 0 denotes a click without purchase and 1 denotes a purchase. For example, means the user clicked on all five products but bought only the first and the last products. Medium-scale experiment (n = 10, 000, k = 5):Similarly to the previous section, we train a two-layer response model that officiates as a new semisynthetic simulation environment. We use the same hyperparameters used previously. FIG9 shows that List-CVAE outperforms all other baseline models within 500 training steps, which corresponds to having seen less than 10 −11 % of all possible slates. Large-scale experiment (n = 1 million, 2 millions, k = 5):We synthesize 1,990k documents by adding independent Gaussian noise N (0, 10 −2 · I) to the original 10k documents and label the synthetic documents by predicted responses from the response model. Thus the new pool of candidate documents consists of 10k original documents and 1,990k synthetic ones, totaling 2 million documents. To match each of the k decoder outputs (x 1, x 2, . . ., x k) with Step 0 Step 0 Step 0 real documents, we uniformly randomly downsample the negative document examples keeping in total only 1000 logits (the dot product outputs in the decoder) during training. At inference time, we pick the argmax for each of k dot products with the full embedding matrix without sampling. This technique speeds up the total training and inference time for 2 million documents to merely 4 minutes on 1 GPU for both the response model (with 40k training steps) and List-CVAE (with 5k training steps). We ran 2 experiments with 1 million and 2 millions document respectively. From the shown in FIG9 and 5d, List-CVAE steadily outperforms all other baselines again. The greatly increased number of training examples helped List-CVAE really learn all the interactions between documents and their respective positional biases. The ing slates were able to receive close to 5 purchases on average due to the limited complexity provided by the response model. In practice, we may not have any close-to-optimal slates in the training data. Hence it is crucial that List-CVAE is able to generalize to unseen optimal conditions. To test its generalization capacity, we use the medium-scale experiment setup on RecSys 2015 dataset and eliminate from the training data all slates where the total user response exceeds some ratio h of the slate size k, i.e. k i=1 r i > hk for h = 80%, 60%, 40%, 20%. FIG10 shows test on increasingly difficult training sets from which to infer on the optimal slates. Without seeing any optimal slates FIG10 ) or slates with 4 or 5 total purchases FIG10 ), List-CVAE can still produce close to optimal slates. Even training on slates with only 0, 1 or 2 total purchases (h = 40%), List-CVAE still surpasses the performance of all greedy baselines within 1000 steps FIG10. Thus demonstrating the strong generalization power of the model. List-CVAE cannot learn much about the 0 1000 2000 3000 4000 5000Step 0 interactions between documents given only 0 or 1 total purchase per slate FIG10 ), whereas the MLP-type models learn purchase probabilities of single documents in the same way as in slates with higher responses. Although evaluation of our model requires choosing the ideal conditioning c at or near the edge of the support of P (c), we can always compromise generalization versus performance by controlling c in practice. Moreover, interactions between documents are governed by similar mechanisms whether they are from the optimal or sub-optimal slates. As the experiment indicate, List-CVAE can learn these mechanisms from sub-optimal slates and generalize to optimal slates. The List-CVAE model moves away from the conventional greedy ranking paradigm, and provides the first conditional generative modeling framework that approaches slate recommendation problem using direct slate generation. By modeling the conditional probability distribution of documents in a slate directly, this approach not only automatically picks up the positional and contextual biases between documents at both training and inference time, but also gracefully avoids the problem of combinatorial explosion of possible slates when the candidate set is large. The framework is flexible and can incorporate different types of conditional generative models. In this paper we showed its superior performance over popular greedy and auto-regressive baseline models with a conditional VAE model. In addition, the List-CVAE model has good scalability. We designed an architecture that uses pretrained document embeddings combined with a negatively downsampled k-head softmax layer that greatly speeds up the training, scaling easily to millions of documents. This test complements the small-scale experiment. To the 100 documents with slate size 10, we add user features into the conditioning c, by adding a set U of 50 different users to the simulation engine (|U| = 50, n = 100, k = 10), permuting the innate attractiveness of documents and their interactions matrix W by a user-specific function π u. Let be the response of the user u on the document d i. During training, the condition c is a concatenation of 16 dimensional user embeddings Θ(u) obtained from the response model, and responses r. At inference time, the model conditions on c = (r, Θ(u)) for each randomly generated test user u. We sweep over hidden layers of 512 or 1024 units in List-CVAE, and all baseline MLP structures. FIG12 show that slates generated by List-CVAE have on average higher clicks than those produced by the greedy baseline models although its convergence took longer to reach than in the small-scale experiment.

represent this text to find a much much longer article about it with say hundreds-thousands of words

Fewshot example: "Adaptive gradient methods, when done right, do not incur a generalization penalty. " == "A commonplace belief in the machine learning community is that using adaptive gradient methods hurts generalization. We re-examine this belief both theoretically and experimentally, in light of insights and trends from recent years. We revisit some previous oft-cited experiments and theoretical accounts in more depth, and provide a new set of experiments in larger-scale, state-of-the-art settings. We conclude that with proper tuning, the improved training performance of adaptive optimizers does not in general carry an overfitting penalty, especially in contemporary deep learning. Finally, we synthesize a ``user's guide'' to adaptive optimizers, including some proposed modifications to AdaGrad to mitigate some of its empirical shortcomings. Adaptive gradient methods have remained a cornerstone of optimization for deep learning. They revolve around a simple idea: scale the step sizes according to the observed gradients along the execution. It is generally believed that these methods enjoy accelerated optimization, and are more robust to hyperparameter choices. For these reasons, adaptive optimizers have been applied across diverse architectures and domains. However, in recent years, there has been renewed scrutiny on the distinction between adaptive methods and "vanilla" stochastic gradient descent (SGD). Namely, several lines of work have purported that SGD, while often slower to converge, finds solutions that generalize better: for the same optimization error (training error), adaptive gradient methods will produce models with a higher statistical error (holdout validation error). This claim, which can be shown to be true in a convex overparameterized examples, has perhaps muddled the consensus between academic research and practitioners pushing the empirical state of the art. For the latter group, adaptive gradient methods have largely endured this criticism, and remain an invaluable instrument in the deep learning toolbox. In this work, we revisit the generalization performance of adaptive gradient methods from an empirical perspective, and examine several often-overlooked factors which can have a significant effect on the optimization trajectory. Addressing these factors, which does not require trying yet another new optimizer, can often account for what appear to be performance gaps between adaptive methods and SGD. Our experiments suggest that adaptive gradient methods do not necessarily incur a generalization penalty: if an experiment indicates as such, there are a number of potential confounding factors and simple fixes. We complete the paper with a discussion of inconsistent evidence for the generalization penalty of adaptive methods, from both experimental and theoretical viewpoints. Our work investigates generalization of adaptive gradient methods, and constructively comments on the following: The brittleness of simple experiments and simple abstractions. We attempt a replication of the experiments from , finding that they have not stood up to unknown hardware and software differences. We show simple theoretical settings where adaptive methods can either fail or succeed dramatically, as compared to SGD. Though each can shed interesting insights, neither abstraction is reflective of the truth. The perils of choosing a large. The innocuous initial accumulator value hyperparameter destroys adaptivity at parameter scales smaller than √. This really matters in large-scale NLP; a foolproof solution is to use our proposed "ε = 0" variant of AdaGrad. The subtleties in conducting a proper optimizer search. The differences between Adam, AdaGrad, and RMSprop are not fundamental; some, like AdaGrad's lack of momentum, are easily fixable. Upon disentangling these differences, and with enough tuning of the learning rate schedule, we suggest that all three are equally good candidates in optimizer search, and can match or beat SGD. Adaptive regularization was introduced along the AdaGrad algorithm in parallel in . A flurry of extensions, heuristics and modifications followed, most notably RMSprop and Adam . Today, these papers have been cited tens of thousands of times, and the algorithms they propose appear in every deep learning framework. For an in-depth survey of the theory of adaptive regularization and its roots in online learning, see . Upon a quick perusal of recent literature, there is plenty of evidence that adaptive methods continue to be relevant in the state of the art. Adam in particular remains a staple in recent developments in fields such as NLP (; ;), deep generative modeling (; ;), and deep reinforcement learning . Adaptive methods have seen adoption in extremely large-scale settings, necessitating modifications to reduce memory consumption (; ; . In recent years, there have been various works attempting to quantify the generalization properties of SGD. These varied perspectives include general analyses based on stability and early stopping , a characterization of the implicit bias in special separable cases (a; b), and a more fine-grained analysis for neural networks exploiting their specific structure . More recently, there has been a growing interest towards understanding the interpolation regime for overparameterized function fitting, where SGD is often the basic object of analysis . Finally, empirical questions on the generalization of adaptive gradient methods were brought to the forefront by , who exhibit empirical and theoretical situations where adaptive methods generalize poorly. Building on this premise, suggest switching from Adam and SGD during training. develop a doctrine of "superconvergence" which eschews adaptive methods. point out some pathological settings where Adam fails to converge, and amends the algorithm accordingly. note some sociological problems leading to misleading research on optimizer selection, providing a benchmarking suite for fairer hyperparameter searches, with mixed preliminary . We begin by reviewing the stochastic optimization setting, and giving rigorous definitions of the adaptive gradient methods commonly used in practice. We will focus on stochastic optimization tasks of the form where the expectation is over a random variable z whose distribution D is initially unknown; in machine learning, z often represents a pair (x, y) of an example x and its corresponding label y, drawn from an unknown population. A stochastic optimization algorithm is given a sample z 1,..., z T ∼ D from the underlying distribution, and produces a pointw ∈ R d whose population loss F (w) is as close as possible to that of the minimizer w = arg min w F (w). Often, iterative (first-order) optimization methods maintain a sequence of iterates w 1,..., w T and, at each step t, use the stochastic gradient to form the next iterate w t+1. The simplest stochastic optimization method is Stochastic Gradient Descent (SGD), whose update rule at step t takes the form where η t > 0 is a step size (or learning rate) parameter, whose scale and time-varying behavior are typically determined via hyperparameter search. Adaptive gradient methods comprise a general family of iterative optimization algorithms which attempt to automatically adapt to anisotropic gradient and parameter sizes. Often, an adaptive method will incorporate a different (adaptively computed) step size for each entry of the gradient vector. More specifically, the most common adaptive methods divide each parameter's gradient update by a second-moment-based estimate of the scale of its historical gradients. A concise way to unify this family of adaptive methods is given by following update equation (starting from an arbitrary initializer w 0): The above update expresses a broad family of methods including SGD, momentum (i.e., Polyak's heavy-ball method), AdaGrad, RMSprop, and Adam. The particular instantiations of the parameters α k, β k are summarized below: Table 1: Parameter settings for common optimization algorithms in the unified framework of Equation 1. Here, 2 denotes the entrywise square of a vector or matrix. We omit the parameters in the adaptive methods, see the discussion in Section 3.1. In this section, we compile some lesser-known practices in the usage of adaptive methods, which we have found to help consistently across large-scale experiments. We emphasize that this collection is restricted to simple ideas, which do not add extraneous hyperparameters or algorithmic alternatives. The general AdaGrad update, as originally proposed by , includes a parameter to allow for convenient inversions. Formally, the update looks like: The inclusion of in the the original proposal of the above updates seems to have been made for convenience of presenting the theoretical . However, in practice, this parameter often turns out to be a parameter that should be tuned depending on the problem instance. The default value of this parameter in the standard implementations of the algorithm tend to be quite high; e.g., in Tensorflow it is 0.1 which can be quite high. As large values would in AdaGrad reducing to SGD with an implicit 1/ √ learning rate, and losing out on all adaptive properties (RMSprop and Adam implementations also have an equivalent epsilon parameter). The effect can be seen in Figure 4 which shows that along many coordinates the accumulated second moments of the gradient are very small, even in the middle of training. At least one work remarks that the ability to choose a large ε in a secondmoment-based adaptive method might be a feature rather than a shortcoming; the smooth interpolation with SGD may improve the stability of more exotic optimizers. This does not appear to be the case for diagonal-matrix adaptive methods, in the NLP setting investigated in this paper. Instead, we suggest removing this hyperparameter altogether and justify it in Section 4.2, and performing the AdaGrad update with the pseudoinverse instead of the full inverse. Then, the update is given by the following: where A † denotes the Moore-Penrose pseudoinverse of A and with the preconditioning matrices updated as before. The above means that if there is a coordinate for which the gradient has been 0 thus far we make no movement in that coordinate. This fix which can similarly be applied to the full matrix version of AdaGrad, does not affect the regret guarantees of AdaGrad. We provide an analysis in the Appendix B, verifying as a sanity check that the standard AdaGrad regret bounds continue to hold when ε is completely removed. A key distinction between AdaGrad, RMSprop and Adam is as follows: AdaGrad does not include momentum, and there is a per-parameter learning rate which is inverse of the accumulated gradient squares for that parameter. RMSprop as described in uses exponential moving averaging rather than straightforward accumulation that AdaGrad relies on, and Adam modifies RMSprop to add momentum for the gradients along with a bias-correction factor. Note that implementation of RMSprop can vary based on the software library; e.g., TensorFlow includes modification to include momentum, while Keras API does not. We note that it is straightforward to extend AdaGrad to incorporate heavy-ball momentum, where we start withḡ 0 = 0 (and from a certain initialization w 0) and iteratively update: The original definition of the Adam optimizer includes a bias correction term, in which the moment estimates are multiplied by the time-varying scalars (1 − β t 1) and (1 − β t 2). As mentioned in the original paper, the bias correction can equivalently be written as an update to the learning rate. In the notation of Table 1: As can be seen from Figure 2, for the typical values of β 1 = 0.9 and β 2 = 0.999, the effective multiplier on the learning rate essentially resembles an external warmup applied on top of the learning rate. The general heuristic of including a warmup phase at the beginning of training has gained significant popularity in state-of-the-art empirical works; see, for example, Applying such a warm up externally on Adam in now 3 hyper-parameters (β 1, β 2 and now the amount of warmup) conflating with each other, making hyper-parameter tuning difficult. Instead we suggest to complete disable this bias correction altogether and use an explicit warmup schedule in place of it. We use such a schedule in all of our experiments for SGD as well as adaptive optimizers as we find that it helps consistently across language modelling experiments. One motivation for warmup during the initial stages of training is that for adaptive updates, the squared norm of the preconditioned gradient during the initial stage is quite large compared to the scale of the parameters. For the initial steps the preconditioned gradient squared norm is proportional to the number of coordinates with non-zero gradients where as the squared norm of the parameters are proportional to the number of nodes. Therefore adaptive methods are naturally forced to start with a smaller learning rate. The warmup in such a case helps the learning rate to rise up while the norm of the gradients fall sharply as training proceeds. Learning rate decay schedules are one of the hardest to tune albeit a crucial hyperparameter of an optimizer. Stochastic gradient like algorithms, domain specific learning rate schedules have been derived over time with a lot of care and effort, examples include Resnet-50 on ImageNet-2012 where state of the art configuration of SGD+Momentum follows a stair-case learning rate schedule (while other type of schedules maybe possible). Adaptive algorithms apriori come with a potential promise of not requiring massive tuning of these schedules as they come with an in-built schedule with the caveat that AdaGrad variants like Adam or RMSprop does not enjoy a data-dependent decay like AdaGrad due to the presence of a non-zero decay factor and requires an external learning rate decay schedule. Even for experiments in which introduces Adam optimizer has experiments to include a 1/ √ T decay schedule for convergence. In our experiments, we found this implicit decay of AdaGrad to be sufficient for achieving superior performance on training a machine translation model, while an external decay rate was necessary for training the Resnet-50 on ImageNet-2012 to high accuracy. We study the empirical performance of various optimization methods for training large state-of-theart deep models, focusing on two domains: natural language processing (machine translation) and image recognition. We study the convergence of various optimization methods when training a Transformer model for machine translation. We used the larger Transformer-Big architecture ; this architecture has 6 layers in the encoder and decoder, with 1024 model dimensions, 8192 hidden dimensions, and 16 attention heads. It was trained on the WMT'14 English to French dataset (henceforth "en→fr") that contains 36.3M sentence pairs. All experiments were carried out on 32 cores of a TPU-v3 Pod and makes use of the Lingvo sequence-to-sequence TensorFlow library. We compared several optimization methods for training; the are reported in Fig. 3. We see that a properly tuned AdaGrad (with ε = 0 and added momentum) outperforms Adam, while SGD with momentum, plain AdaGrad and RMSprop perform much worse on this task. These illustrate that adaptivity and momentum are both extremely effective in training these models. In Section 3.1, we proposed an "ε = 0" variant of AdaGrad. Here we empirically motivate this modification, by investigating the effect of the parameter ε on the performance of AdaGrad. We train the Transformer model from above on the en→fr dataset using AdaGrad while varying the value of ε. The are given in Fig. 4. We see drastic improvement in convergence as we lower the value of ε down to 10 −7 (lower values do not improve convergence further and are thus omitted from the figure). To see where these dramatic improvements come from, we also visualize in?? the histogram of the square gradient values for the embedding layer of the model at step t = 116200, which indicates that a large fraction of the cumulative gradient entries have extremely small magnitudes. The choice of ε is thus important, and justify our prescription of removing the dependency all-together instead of tuning it as a separate hyper-parameter. Next, we trained a ResNet-50 architecture on the Imagenet-2012 dataset. The task is to classify images as belonging to one of the 1000 classes. Our training setup consists of 512 cores of a TPU v3 Pod and makes use of a relatively large batch size of 16386. As a baseline, we considered SGD with momentum with a highly-tuned staircase learning rate schedule, that achieves 75.3% test accuracy after 90 epochs. We compared several optimization methods on this task as seen in Fig. 5: the straightforward application of AdaGrad (with a fixed ε and with heavy ball momentum) achieves only a paltry 63.94% test accuracy. Noticing that AdaGrad implicit decay schedule does not decay sufficiently fast, an external decay rate was added starting at epoch 50 of the form (1 − current epoch -50 50) 2. This change was sufficient for AdaGrad to reach a test accuracy of 74.76%-a drastic >∼ 10% jump. As demonstrated, learning rate schedule is a highly important hyperparameter and requires tuning for each task. E.g., the baseline SGD is highly tuned and follows an elaborate stair case learning rate to reach 75% test accuracy. We attempted to reproduce the experiments from , using the same codebases and identical hyperparameter settings. Although we were able to replicate some of their findings on these smaller-scale experiments, others appear to be sensitive to hyperparameter tuning, and perhaps subtle changes in the deep learning software and hardware stack that have occurred during the two years since the publication of that paper. In this section, we summarize these findings. Image classification. On the classic benchmark task of CIFAR-10 classification with a VGG network , we were able to replicate the perfectly, using the same codebase 2. We repeated the hyperparameter search reported in the paper, found the same optimal base learning rates for each optimizer, and found the same stratification in performance between non-adaptive methods, Adam & RMSprop, and AdaGrad. Character-level language modeling. Curiously, our replication of the language modeling experiment using the same popular repository 3 was successful in reproducing the optimal hyperparameter settings, but ed in an opposite . Here, SGD found the objective with the smallest training loss, but Adam exhibited the best generalization performance. We believe that software version discrepancies (our setup: CUDA 10.1, cuDNN 7.5.1) may account for these small differences. Generative parsing. We turn to the Penn Treebank constituency parsing code 4 accompanying . Using the same architectural and training protocol modifications as specified in , we were able to get the model to converge with each optimizer. However, for two of the settings (SGD and RMSprop), the best reported learning rates exhibited non-convergence (the fainter curves in Figure 6). Similarly as the above experiment, the ranking of optimizers' training and generalization performance differs from that seen in the original report. include a fourth set of experiments, generative parsing of Penn Treebank, using the code 5 accompanying . Unfortunately, this DyNet implementation, which was last updated in 2016, encountered a fatal memory leak when training with our DyNet 2.1 setup. All relevant plots are given in Figure 6, with color codes selected to match Figures 1 and 2 in. Together, these experiments are further evidence for a widespread reproducibility crisis in deep learning: despite the authors' exceptional transparency in disclosing their optimizer selection and evaluation protocol, these benchmarks have turned out to be brittle for unknown reasons. Along the same lines as the random-seed-tuning experiments of , this suggests that there are further technical complications to the problems of credible optimizer evaluation addressed by , even on well-known supervised learning benchmarks. Character-level language modeling with a 2-layer LSTM. The original reported hyperparameters are the best and all optimizers converge to reasonable solutions, but contradictory about generalization arise. Bottom: 3-layer LSTM for generative parsing. Training does not converge with all reported learning rates; about generalization are unclear. In this section we provide two simple examples of stochastic convex problems where it can be seen that when it comes to generalization both AdaGrad and SGD can be significantly better than the other depending on the instance. Our purpose to provide both the examples is to stress our point that the issue of understanding the generalization performance of SGD vs. adaptive methods is more nuanced than what simple examples might suggest and hence such examples should be treated as qualitative indicators more for the purpose of providing intuition. Indeed which algorithm will perform better on a given problem, depends on various properties of the precise instance. We provide a brief intuitive review of the construction provided by; for a precise description, see Section 3.3 of that paper. Consider a setting of overparameterized linear regression, where the true output (i.e. dependent variable) y ∈ {±1} is the first coordinate of the feature vector (independent variable) x. The next two coordinates of x are "dummy" coordinates set to 1; then, the coordinates are arranged in blocks which only appear once per sample, taking the value of y. The key idea is that in this setting, the solution space that AdaGrad explores is always in the subspace of the sign vector of X y. As a , AdaGrad treats the first three coordinates essentially indistinguishably putting equal mass on each. It can then be seen that for any new example the AdaGrad solution does not extract the true label information from the first three coordinates and hence gets the prediction wrong, leading to high generalization error; the other distinguishing features belong to the new unique block which are set to 0 for the AdaGrad solution, as it has not seen them. This example is motivated from the original AdaGrad paper , adapted to the overparameterized setting. Consider a distribution Z supported over {0, 1} d with equal 1/d mass over vectors with exactly one 1 and 0 mass everywhere else. Let the label distribution be always y = 1. Consider sampling a dataset S of size c · d where c ≤ 1 (corresponding to the overparameterized setting) and consider the hinge loss f t (x) = [1 − y t (z t x t)] + where (z t, y t) denotes the t-th (example, label) pair. Note that there is an optimal predictor given by the all-ones vector. Running AdaGrad in such a setting, it can be seen that the first time a vector that has not appeared yet is sampled, AdaGrad quickly adapts by setting the coordinate corresponding to the vector to 1 and thereby making 0 error on the example. Therefore after one epoch of AdaGrad (cd steps), the training error reduces to 0 and the average test error becomes roughly (1 − c). On the other hand, for SGD (with an optimal 1/ √ t decay scheme) after say cd/2 steps, the learning rate reduces to at most O(1/ √ d) and therefore in the next cd/2 steps SGD reduces the error at most by a factor of O(1 − 1 √ d), leading to a total test error of at least ∼ (1 − c/2) after a total of cd steps. This is significantly smaller than the error achieved by AdaGrad at this stage. Further note that to get down to the same test error as that achieved by AdaGrad, it can be seen that SGD requires at least Ω(√ d) times more steps than AdaGrad."

We used a CVAE type model structure to learn to directly generate slates/whole pages for recommendation systems.

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent the scientific article to find a short TLDR

Question: Neural networks for structured data like graphs have been studied extensively in recent years. To date, the bulk of research activity has focused mainly on static graphs. However, most real-world networks are dynamic since their topology tends to change over time. Predicting the evolution of dynamic graphs is a task of high significance in the area of graph mining. Despite its practical importance, the task has not been explored in depth so far, mainly due to its challenging nature. In this paper, we propose a model that predicts the evolution of dynamic graphs. Specifically, we use a graph neural network along with a recurrent architecture to capture the temporal evolution patterns of dynamic graphs. Then, we employ a generative model which predicts the topology of the graph at the next time step and constructs a graph instance that corresponds to that topology. We evaluate the proposed model on several artificial datasets following common network evolving dynamics, as well as on real-world datasets. Results demonstrate the effectiveness of the proposed model. Graph neural networks (GNNs) have emerged in recent years as an effective tool for analyzing graph-structured data (; ; ;). These architectures bring the expressive power of deep learning into non-Euclidean data such as graphs, and have demonstrated convincing performance in several graph mining tasks, including graph classification , link prediction , and community detection ). So far, GNNs have been mainly applied to tasks that involve static graphs. However, most real-world networks are dynamic, i. e., nodes and edges are added and removed over time. Despite the success of GNNs in various applications, it is still not clear if these models are useful for learning from dynamic graphs. Although some models have been applied to this type of data, most studies have focused on predicting a low-dimensional representation (i. e., embedding) of the graph for the next time step (; ; ; ;). These representations can then be used in downstream tasks (; ; ;). Predicting the topology of the graph is a task that has not been properly addressed yet. Graph generation, another important task in graph mining, has attracted a lot of attention from the deep learning community in recent years. The objective of this task is to generate graphs that exhibit specific properties, e. g., degree distribution, node triangle participation, community structure etc. Traditionally, graphs are generated based on some network generation model such as the Erdős-Rényi model. These models focus on modeling one or more network properties, and neglect the others. Neural network approaches, on the other hand, can better capture the properties of graphs since they follow a supervised approach (; ;). These architectures minimize a loss function such as the reconstruction error of the adjacency matrix or the value of a graph comparison algorithm. Capitalizing on recent developments in neural networks for graph-structured data and graph generation, we propose in this paper, to the best of our knowledge, the first framework for predicting the evolution of the topology of networks in time. The proposed framework can be viewed as an encoderdecoder architecture. The "encoder" network takes a sequence of graphs as input and uses a GNN to produce a low-dimensional representation for each one of these graphs. These representations capture structural information about the input graphs. Then, it employs a recurrent architecture which predicts a representation for the future instance of the graph. The "decoder" network corresponds to a graph generation model which utilizes the predicted representation, and generates the topology of the graph for the next time step. The proposed model is evaluated over a series of experiments on synthetic and real-world datasets. To measure its effectiveness, the generated graphs need to be compared with the corresponding ground-truth graph instances. To this end, we use the Weisfeiler-Lehman subtree kernel which scales to very large graphs and has achieved state-of-the-art on many graph datasets . The proposed model is compared against several baseline methods. Results show that the proposed model is very competitive, and in most cases, outperforms the competing methods. The rest of this paper is organized as follows. Section 2 provides an overview of the related work and elaborates our contribution. Section 3 introduces some preliminary concepts and definitions related to the graph generation problem, followed by a detailed presentation of the components of the proposed model. Section 4 evaluates the proposed model on several tasks. Finally, Section 5 concludes. Our work is related to random graph models. These models are very popular in graph theory and network science. The Erdős-Rényi model (Erdős & Rényi, 1960), the preferential attachment model (Albert & Barabási, 2002), and the Kronecker graph model are some typical examples of such models. To predict how a graph structure will evolve over time, the values of the parameters of these models can be estimated based on the corresponding values of the observed graph instances, and then the estimated values can be passed on to these models to generate graphs. Other work along a similar direction includes neural network models which combine GNNs with RNNs (; ;). These models use GNNs to extract features from a graph and RNNs for sequence learning from the extracted features. Other similar approaches do not use GNNs, but they instead perform random walks or employ deep autoencoders . All these works focus on predicting how the node representations or the graph representations will evolve over time. However, some applications require predicting the topology of the graph, and not just its low-dimensional representation. The proposed model constitutes the first step towards this objective. In this Section, we first introduce basic concepts from graph theory, and define our notation. We then present EvoNet, the proposed framework for predicting the evolution of graphs. Since the proposed model comprises of several components, we describe all these components in detail. Let G = (V, E) be an undirected, unweighted graph, where V is the set of nodes and E is the set of edges. We will denote by n the number of vertices and by m the number of edges. We define a permutation of the nodes of G as a bijective function π: V → V, under which any graph property of G should be invariant. We are interested in the topology of a graph which is described by its adjacency matrix A π ∈ R n×n with a specific ordering of the nodes π 1. Each entry of the adjacency matrix is defined as In what follows, we consider the "topology", "structure" and "adjacency matrix" of a graph equivalent to each other. In many real-world networks, besides the adjacency matrix that encodes connectivity information, nodes and/or edges are annotated with a feature vector, which we denote as X ∈ R n×d and L ∈ R m×d, respectively. Hence, a graph object can be also written in the form of a triplet G = (A, X, L). In this paper, we use this triplet to represent all graphs. If a graph does not contain node/edge attributes, we assign attributes to it based on local properties (e. g., degree, k-core number, number of triangles, etc). An evolving network is a graph whose topology changes as a function of time. Interestingly, almost all real-world networks evolve over time by adding and removing nodes and/or edges. For instance, in social networks, people make and lose friends over time, while there are people who join the network and others who leave the network. An evolving graph is a sequence of graphs {G 0, G 1, . . ., G T} where G t = (A t, X t, E t) represents the state of the evolving graph at time step t. It should be noted that not only nodes and edges can evolve over time, but also node and edge attributes. However, in this paper, we keep node and edge attributes fixed, and we allow only the node and edge sets of the graphs to change as a function of time. The sequence can thus be written as {G t = (A t, X, E)} t∈ [0,T]. We are often interested in predicting what "comes next" in a sequence, based on data encountered in previous time steps. In our setting, this is equivalent to predicting G t based on the sequence {G k} k<t. In sequential modeling, we usually do not take into account the whole sequence, but only those instances within a fixed small window of size w before G t, which we denote as {G t−w, G t−w+1, . . ., G t−1}. We refer to these instances as the graph history. The problem is then to predict the topology of G t given its history. The proposed architecture is very similar to a typical sequence learning framework. The main difference lies in the fact that instead of vectors, in our setting, the elements of the sequence correspond to graphs. The combinatorial nature of graph-structured data increases the complexity of the problem and calls for more sophisticated architectures than the ones employed in traditional sequence learning tasks. Specifically, the proposed model consists of three components: a graph neural network (GNN) which generates a vector representation for each graph instance, a recurrent neural network (RNN) for sequential learning, and a graph generation model for predicting the graph topology at the next time step. This framework can also be viewed as an autoencoder-decoder model. The first two components correspond to an encoder network which maps the sequence of graphs into a sequence of vectors and predicts a representation for the next in the sequence graph. The decoder network consists of the last component of the model, and transforms the above representation into a graph. Figure 1 illustrates the proposed model. In what follows, we present the different components of EvoNet in detail. Graph Neural Networks (GNNs) have recently emerged as a dominant paradigm for performing machine learning tasks on graphs. Several GNN variants have been proposed in the past years. All these models employ some message passing procedure to update node representations. Specifically, each node updates its representation by aggregating the representations of its neighbors. After k iterations of the message passing procedure, each node obtains a feature vector which captures the structural information within its k-hop neighborhood. Then, GNNs compute a feature vector for the entire graph using some permutation invariant readout function such as summing the representations of all the nodes of the graph. As described below, the learning process can be divided into three phases: aggregation, update, and readout. Aggregation In this phase, the network computes a message for each node of the graph. To compute that message for a node, the network aggregates the representations of its neighbors. Formally, at time t + 1, a message vector m t+1 v is computed from the representations of the neighbors where AGGREGATE is a permutation invariant function. Furthermore, for the network to be end-toend trainable, this function needs to be differentiable. In our case, AGGREGATE is a multi-layer perceptron (MLP) followed by a sum function. h 2) The UPDATE function also needs to be differentiable. To combine the two feature vectors (i. e., h t v and m t+1 v), we have employed the Gated Recurrent Unit proposed in : Omitting biases for readability, we have: where the W and U matrices are trainable weight matrices, σ is the sigmoid function, and r v and z v are the parameters of the reset and update gates for a given node. Readout The Aggregation and Update steps are repeated for T time steps. The emerging node representations {h T v} v∈V are aggregated into a single vector which corresponds to the graph representation, as follows: ) where READOUT is a differentiable and permutation invariant function. This vector captures the topology of the input graph. To generate h G, we utilize Set2Set . Other functions such as the sum function were also considered, but were found less effective in preliminary experiments. Given an input sequence of graphs, we use the GNN described above to generate a vector representation for each graph in the sequence. Then, to process this sequence, we use a recurrent neural network (RNN). RNNs use their internal state (i. e., memory) to preserve sequential information. These networks exhibit temporal dynamic behavior, and can find correlations between sequential events Specifically, an RNN processes the input sequence in a series of time steps (i. e., one for each element in the sequence). For a given time step t, the hidden state h t at that time step is updated as: where f is a non-linear activation function. A generative RNN outputs a probability distribution over the next element of the sequence given its current state h t. RNNs can be trained to predict the next element (e. g., graph) in the sequence, i. e., it can learn the conditional distribution p(G t |G 1, . . ., G t−1). In our implementation, we use a Long Short-Term Memory (LSTM) network that reads sequentially the vectors {h Gi} i∈[t−w,t−1] produced by the GNN, and generates a vector h G T that represents the embedding of G T, i. e., the graph at the next time step. The embedding incorporates topological information and will serve as input to the graph generation module. Along with the GNN component, this architecture can be seen as a form of an encoder network. This network takes as input a sequence of graphs and projects them into a low-dimensional space. To generate a graph that corresponds to the evolution of the current graph instance, we capitalize on a recently-proposed framework for learning generative models of graphs . This framework models a graph in an autoregressive manner (i. e., a sequence of additions of new nodes and edges) to capture the complex joint probability of all nodes and edges in the graph. Formally, given a node ordering π, it considers a graph G as a sequence of vectors: where S π i = [a 1,i, . . ., a i−1,i] ∈ {0, 1} i−1 is the adjacency vector between node π(i) and the nodes preceding it ({π,..., π(i − 1)}). We adapt this framework to our supervised setting. The objective of the generative model is to maximize the likelihood of the observed graphs of the training set. Since a graph can be expressed as a sequence of adjacency vectors (given a node ordering), we can consider instead the distribution p(Ŝ π ; θ), which can be decomposed in an autoregressive manner into the following product: This product can be parameterized by a neural network. Specifically, following , we use a hierarchical RNN consisting of two levels: the graph-level RNN which maintains the state of the graph and generates new nodes and thus learns the distribution p(Ŝ π i |Ŝ π k:k<i) and the edge-level RNN which generates links between each generated node and previously-generated nodes and thus learns the distribution p(â π ji |â π li:l<j). More formally, we have: where h i is the state vector of the graph-level RNN (i. e., RNN 1) that encodes the current state of the graph sequence and is initialized by h G T, the predicted embedding of the graph at the next time step T. The output of the graph-level RNN corresponds to the initial state of the edge-level RNN (i. e., RNN 2). The ing value is then squashed by a sigmoid function to produce the probability of existence of an edgeâ j,i. In other words, the model learns the probability distribution of the existence of edges and a graph can then be sampled from this distribution, which will serve as the predicted topology for the next time step T. To train the model, the cross-entropy loss between existence of each edge and its probability of existence is minimized: Node ordering It should be mentioned that node ordering π has a large impact on the efficiency of the above generative model. Note that a good ordering can help us avoid the exploration of all possible node permutations in the sample space. Different strategies such as the Breadth-First-Search ordering scheme can be employed to improve scalability . However, in our setting, the nodes are distinguishable, i. e., node v of G i and node v of G i+1 correspond to the same entity. Hence, we can impose an ordering onto the nodes of the first instance of our sequence of graphs, and then utilize the same node ordering for the graphs of all subsequent time steps (we place new nodes at the end of the ordering). In this Section, we evaluate the performance of EvoNet on synthetic and real-world datasets for predicting the evolution of graph topology, and we compare it against several baseline methods. We use both synthetic and real-world datasets. The synthetic datasets consist of sequences of graphs where there is a specific pattern on how each graph emerges from the previous graph instance, i. e., add/remove some graph structure at each time step. The real-world datasets correspond to single graphs whose nodes incorporate temporal information. We decompose these graphs into sequences of snapshots based on their timestamps. We fix the length of the sequences to 1000 time steps. The size of the graphs in each sequence ranges from tens of nodes to several thousand of nodes. Path graph A path graph can be drawn such that all vertices and edges lie on a straight line. We denote a path graph of n nodes as P n. In other words, the path graph P n is a tree with two nodes of degree 1, and the other n − 2 nodes of degree 2. We consider two scenarios. In both cases the first graph in the sequence is P 3. In the first scenario, at each time step, we add one new node to the previous graph instance and we also add an edge between the new node and the last according to the previous ordering node. The second scenario follows the same pattern, however, every three steps, instead of adding a new node, we remove the first according to the previous ordering node (along with its edge). Cycle graph A cycle graph C n is a graph on n nodes containing a single cycle through all the nodes. Note that if we add an edge between the first and the last node of P n, we obtain C n. Similar to the above case, we use C 3 as the first graph in the sequence, and we again consider two scenarios. In the first scenario, at each time step, we increase the size of the cycle, i. e., from C i, we obtain C i+1 by adding a new node and two edges, the first between the new node and the first according to the previous ordering node and the second between the new node and the last according to the previous ordering node. In the second scenario, every three steps, we remove the first according to the ordering node (along with its edges), and we add an edge between the second and the last according to the ordering nodes. Ladder graph The ladder graph L n is a planar graph with 2n vertices and 3n − 2 edges. It is the cartesian product of two path graphs, as follows: L n = P n × P 2. As the name indicates, the ladder graph L n can be drawn as a ladder consisting of two rails and n rungs between them. We consider the following scenario: at each time step, we attach one rung (P 2) to the tail of the ladder (the two nodes of the rung are connected to the two last according to the ordering nodes). For all graphs, we set the attribute of each node equal to its degree, while we set the attribute of all edges to the same value (e. g., to 1). Besides synthetic datasets, we also evaluate EvoNet on real-world datasets. These datasets contain graphs derived from the Bitcoin transaction network, a who-trust-whom network of people who trade using Bitcoin (; . Due to the anonymity of Bitcoin users, platforms seek to maintain a record of users' reputation in Bitcoin trades to avoid fraudulent transactions. The nodes of the network represent Bitcoin users, while an edge indicates that a trade has been executed between its two endpoint users. Each edge is annotated with an integer between −10 and 10, which indicates the rating of the one user given by the other user. The network data are collected separately from two platforms: Bitcoin OTC 2 and Bitcoin Alpha 3 . More details about these two datasets are given in Table 1 . For all graphs, we set the attribute of each node equal to the average rating that the user has received from the rest of the community, and the attribute of each edge equal to the rating between its two endpoint users. We compare EvoNet against several random graph models: the Erdős-Rényi model (Erdős & Rényi, 1960), the Stochastic Block model , Barabási-Albert model (Albert & Barabási, 2002), and the Kronecker Graph model . These are the traditional methods to study the topology evolution of temporal graphs, by proposing a driven mechanism behind the evolution. In general, it is very challenging to measure the performance of a graph generative model since it requires comparing two graphs to each other, a long-standing problem in mathematics and computer science . We propose to use graph kernels to compare graphs to each other, and thus to evaluate the quality of the generated graphs. Graph kernels have emerged as one of the most effective tools for graph comparison in recent years . A graph kernel is a symmetric positive semidefinite function which takes two graphs as input, and measures their similarity. In our experiments, we employ the Weisfeiler-Lehman subtree kernel which counts label-based subtree-patterns . Note that we also normalize the kernel values, and thus the emerging values lie between 0 and 1. As previously mentioned, each dataset corresponds to a sequence of graphs where each sequence represents the evolution of the topology of a single graph in 1000 time steps. We use the first 80% of these graph instances for training and the rest of them serve as our test set. The window size w is set equal to 10, which means that we feed 10 consecutive graph instances to the model and predict the topology of the instance that directly follows the last of these 10 input instances. Each graph of the test set along with its corresponding predicted graph is then passed on to the Weisfeiler-Lehman subtree kernel which measures their similarity and thus the performance of the model. The hyperparameters of EvoNet are chosen based on its performance on a validation set. The parameters of the random graph models are set under the principle that the generated graphs need to share similar properties with the ground-truth graphs. For instance, in the case of the Erdős-Rényi model, the probability of adding an edge between two nodes is set to some value such that the density of the generated graph is identical to that of the ground-truth graph. However, since the model should not have access to such information (e. g., density of the ground-truth graph), we use an MLP to predict this property based on past data (i. e., the number of nodes and edges of the previous graph instances). This is in par with how the proposed model computes the size of the graphs to be generated (i. e., using also an MLP). Figure 2 illustrates the experimental on the synthetic datasets. Since the graph structures contained in the synthetic datasets are fairly simple, it is easy for the model to generate graphs very similar to the ground-truth graphs (normalized kernel values > 0.9). Hence, instead of reporting the kernel values, we compare the size of the predicted graphs against that of the ground-truth graphs. The figures visualize the increase of graph size on real sequence (orange) and predicted sequence (blue). For path graphs, in spite of small variance, we have an accurate prediction on the graph size. For ladder graph, we observe a mismatch at the beginning of the sequence for small size graphs but then a coincidence of the two lines on large size graphs. This mismatch on small graphs may be due to a more complex structure in ladder graphs such as cycles, as supported by the of cycle graph on the right figure, where we completely mispredict the size of cycle graphs. In fact, we fail to reconstruct the cycle structure in the prediction, with all the predicted graphs being path graphs. This failure could be related to the limitations of GNN mentioned in Real-World datasets Finally, we analyze the performance of our model on real datasets: the Bitcoin-OTC and Bitcoin-Alpha. We obtain the similarities between each pair of real and predicted graphs in the sequence and draw a histogram to illustrate the distribution of similarities. The are shown in Figure 3 respectively for the two datasets. Among all the traditional random graph models, Kronecker graph model (with learnable parameter) performs the best, however on both datasets, our proposed method EvoNet (in blue) outperforms tremendously all other methods, with an average similarity of 0.82 on BTC-OTC dataset and 0.55 on BTC-Alpha dataset. Detailed statistics can be found in Table 2. Overall, our experiments demonstrate the advantage of EvoNet over the traditional random graph models on predicting the evolution of dynamic graphs. In this paper, we proposed EvoNet, a model that predicts the evolution of dynamic graphs, following an encoder-decoder framework. We also proposed an evaluation methodology for this task which capitalizes on the well-established family of graph kernels. Experiments show that the proposed model outperforms traditional random graph methods on both synthetic and real-world datasets.

Represent the TLDR for finding the corresponding scientific article.

Combining graph neural networks and the RNN graph generative model, we propose a novel architecture that is able to learn from a sequence of evolving graphs and predict the graph topology evolution for the future timesteps

Represent this text to find a much much longer article about it with say hundreds-thousands of words