Question: Represent the scientific article to find a short TLDR:

Question: We propose and study a method for learning interpretable representations for the task of regression. Features are represented as networks of multi-type expression trees comprised of activation functions common in neural networks in addition to other elementary functions. Differentiable features are trained via gradient descent, and the performance of features in a linear model is used to weight the rate of change among subcomponents of each representation. The search process maintains an archive of representations with accuracy-complexity trade-offs to assist in generalization and interpretation. We compare several stochastic optimization approaches within this framework. We benchmark these variants on 100 open-source regression problems in comparison to state-of-the-art machine learning approaches. Our main finding is that this approach produces the highest average test scores across problems while producing representations that are orders of magnitude smaller than the next best performing method (gradient boosting). We also report a negative in which attempts to directly optimize the disentanglement of the representation in more highly correlated features. The performance of a machine learning (ML) model depends primarily on the data representation used in training BID3, and for this reason the representational capacity of neural networks (NN) is considered a central factor in their success in many applications BID19. To date, there does not seem to be a consensus on how the architecture should be designed. As problems grow in complexity, the networks proposed to tackle them grow as well, leading to an intractable design space. One design approach is to tune network architectures through network hyperparameters using grid search or randomized search BID4 with cross validation. Often some combination of hyperparameter tuning and manual design by expertise/intuition is done BID19. Many approaches to network architecture search exist, including weight sharing BID53 and reinforcement learning BID70. Another potential solution explored in this work (and others) is to use population-based stochastic optimization (SO) methods, also known as metaheuristics BID44. In SO, several candidate solutions are evaluated and varied over several iterations, and heuristics are used to select and update the candidate networks until the population produces a desirable architecture. The general approach has been studied at least since the late 80s in various forms BID45 BID69 BID60 for NN design, with several recent applications BID55 BID28 BID9 BID54.In practice, the adequacy of the architecture is often dependent on conflicting objectives. For example, interpretability may be a central concern, because many researchers in the scientific community rely on ML models not only to provide predictions that match data from various processes, but to provide insight into the nature of the processes themselves. Approaches to interpretability can be roughly grouped into semantic and syntactic approaches. Semantic approaches encompass methods that attempt to elucidate the behavior of a model under various input conditions as a way of explanation (e.g. BID56). Syntactic methods instead focus on the development of concise models that offer insight by virtue of their simplicity, in a similar vein to models built from first-principles (e.g. BID63 BID57). Akin to the latter group, our goal is to discover the simplest description of a process whose predictions generalize as well as possible. Good representations should also disentangle the factors of variation BID3 in the data, in order to ease model interpretation. Disentanglement implies functional modularity; i.e., sub-components of the network should encapsulate behaviors that model a sub-process of the task. In this sense, stochastic methods such as evolutionary computation (EC) appear well-motivated, as they are premised on the identification and propagation of building blocks of solutions BID23. Experiments with EC applied to networks suggest it pressures networks to be modular BID24 BID29. Although the identification functional building blocks of solutions sounds ideal, we have no way of knowing a priori whether a given problem will admit the identification of building blocks of solutions via heuristic search BID49. Our goal in this paper is thus to empirically assess the performance of several SO approaches in a system designed to produce intelligible representations from NN building blocks for regression. In Section 2, we introduce a new method for optimizing representations that we call the feature engineering automation tool (FEAT) 1. The purpose of this method is to optimize an archive of representations that characterize the trade-off between conciseness and accuracy among representations. Algorithmically, two aspects of the method distinguish FEAT from previous work. First, it represents the internal structure of each NN as a set of syntax trees, with the goal of improving the transparency of the ant representations. Second, it uses weights learned via gradient descent to provide feedback to the variation process at a more granular level. We compare several multi-objective variants of this approach using EC and non-EC methods with different sets of objectives. We discuss related work in more detail in Section 3. In section 4 and 5, we describe and conduct an experiment that benchmarks FEAT against state-of-the-art ML methods on 100 open-source regression problems. Future work based on this analysis is discussed in Section 6, and additional detailed are provided in the Appendix. We are interested in the task of regression, for which the goal is to build a predictive modelŷ(x) using DISPLAYFORM0. The regression modelŷ(x) associates the inputs x ∈ R d with a real-valued output y ∈ R. The goal of feature engineering / representation learning is to find a new representation of x via a m-dimensional feature mapping φ(x): R d → R m, such that a model y(φ(x)) outperforms the modelŷ(x). We will assume that each predictor in x is scaled to zero mean, unit-variance. When applying a NN to a traditional ML task like regression or classification, a fixed NN architecture φ(x, θ), parameterized by θ, is chosen and used to fit a model DISPLAYFORM1 T is a NN representation with m nodes in the final hidden layer and a linear output layer with estimated coefficientsβ = [β 1 . . .β m]T. Typically the problem is then cast as a parameter optimization problem that minimizes a loss function via gradient descent. In order to tune the structure of the representation, we instead wish to solve the joint optimization problem DISPLAYFORM2 whereφ(x,θ) is chosen to minimize a cost function L, with global optimum φ * (x, θ *). (L may depend on θ and β in the case of regularization.) S is the space of possible representations realizable by the search procedure, and φ * is the true structure of the process underlying the data. The assumption of SO approaches such as evolutionary computation (EC) and simulated annealing (SA) BID11. FEAT uses a typical µ + λ evolutionary updating scheme, where µ = λ = P. The method optimizes a population of potential representations, N = {n 1 . . . n P}, where n is an "individual" in the population, iterating through these steps:1. Fit a linear modelŷ = x Tβ. Create an initial population N consisting of this initial representation, φ = x, along with P − 1 randomly generated representations that sample x proportionally toβ. (a) Select parents P ⊆ N using a selection algorithm. (b) Apply variation operators to parents to generate P offspring O; N = N ∪ O (c) Reduce N to P individuals using a survival algorithm.3. Select and return n ∈ N with the lowest error on a hold-out validation set. Individuals are evaluated using an initial forward pass, after which each representation is used to fit a linear model (Eqn. 1) using ridge regression BID22. The weights of the differentiable features in the representation are then updated using stochastic gradient descent. The salient aspect of the proposed method is its use of syntax trees to represent the internal architecture of the network, as shown in FIG0. FEAT constructs these trees from elementary boolean-and continuousvalued functions and literals (see TAB0). This scheme is inspired by symbolic regression (SR) BID34. In contrast to typical SR, each individual n is a set of such trees, the output of which is interpreted as a candidate representation, i.e. φ(x) = [φ 1 . . . φ m] for an individual with m trees. The second difference from traditional SR is that the weights of differentiable nodes are encoded in the edges of the graph, rather than as independent nodes. We include instructions typically used as activation functions used in NN, e.g. tanh, sigmoid, logit and relu nodes, elementary arithmetic and boolean operators. Although a fully connected feedforward NN could be represented by this construction, representations in FEAT are biased to be thinly connected. Our hypothesis is that by doing so, we will improve the representation's legibility without sacrificing its capacity for modelling nonlinear relationships. During variation, the representations are perturbed using a set of mutation and crossover methods. FEAT chooses among 6 variation operators that are as follows. Point mutation changes a node type to a random one with matching output type and arity. Insert mutation replaces a node with a randomly generated depth 1 subtree. Delete mutation removes a feature or replaces a sub-program with an input node, with equal probability. Insert/Delete dimension adds or removes a new feature. Sub-tree crossover replaces a sub-tree from one parent with the sub-tree of another parent. Dimension crossover swaps two features between parents. The exact probabilities of each variation operator will affect the performance of the algorithm, and others have proposed methods for adjusting these probabilities, e.g. BID26. For the purposes of our study, we use each operator with uniform probability. Feedback The use of an ML model to assess the fitness of each representation can be used to provide information about the elements of the representation that should be changed. In particular, we assume that programs in the representation with small coefficients are the best candidates for mutation and crossover. With this in mind, let n be an m-dimensional candidate representation with associated coefficients DISPLAYFORM0 The probability of mutation for tree i in n is denoted P M i (n), and defined as follows: DISPLAYFORM1 The normalized coefficient magnitudesβ ∈ are used to define softmax-normalized probabilities, s in Eqn. 3. The smaller the coefficient, the higher the probability of mutation. The parameter f is used to control the amount of feedback used to weight the probabilities; 1 m in this case represents uniform probability. Among nodes in tree m, mutation occurs with uniform probability. This weighting could be extended for differentiable nodes by weighting the within-tree probabilities by the magnitude of the weights associated with each node. However we expect this would yield diminishing returns. The selection step selects among P parents those representations that will be used to generate offspring. Following variation, the population consists of 2P representations of parents and offspring. The survival step is used to reduce the population back to size P, at which point the generation is finished. In our initial study, we empirically compared five algorithms for selection and survival: 1) -lexicase selection (Lex) BID39, 2) non-dominated sorting genetic algorithm (NSGA2) BID11, 3) a novel hybrid algorithm using Lex for selection and NSGA2 for survival, 4) simulated annealing BID31, and 5) random search. These comparisons are described in Appendix Section A.2. We found that the hybrid algorithm performed the best; it is described below. Parents are selected using Lex. Lex was proposed for regression problems BID39 BID40 as an adaption of lexicase selection BID58 for continuous domains. Under -lexicase selection, parents are chosen by filtering the population according to randomized orderings of training samples with the threshold defined relative to the sample loss among the selection pool. This filtering strategy scales probability of selection for an individual based on the difficulty of the training cases the individual performs well on. Lex has shown strong performance among SR methods in recent tests, motivating our interest in studying it BID50. The survival step for Lex just preserves offspring plus the best individual in the population. Survival is conducted using the survival sub-routine of NSGA2, a popular strategy for multi-objective optimization BID11. NSGA2 applies preference for survival using Pareto dominance relations. An individual (n i) is said to dominate another (n j) if, for all objectives, n i performs at least as well as n j, and for at least one objective, n i strictly outperforms n j. The Pareto front is the set of individuals in N that are non-dominated in the population and thus represent optimal trade-offs between objectives found during search. Individuals are assigned a Pareto ranking that specifies the number of individuals that dominate them, thereby determining their proximity to the front. The survival step of NSGA2 begins by sorting the population according to their Pareto front ranking and choosing the lowest ranked individuals for survival. To break rank ties, NSGA2 assigns each individual a crowding distance measure, which quantifies an individual's distance to its two adjacent neighbors in objective space. If a rank level does not completely fit in the survivor pool, individuals of that rank are sorted by highest crowding distance and added in order until P individuals are chosen. We consider three objectives in our study corresponding to three goals: first, to reduce model error; second, to minimize complexity of the representation; and third, to minimize the entanglement of the representation. We test the third objective using two different metrics: the correlation of the transformation matrix φ(x) and its condition number. These metrics are defined below. The first objective always corresponds to the mean squared loss function for individual n, and the second corresponds to the complexity of the representation. There are many ways to define complexity of an expression; one could simply look at the number of operations in a representation, or look at the behavioral complexity of the representation using a polynomial order BID66 ). The one we use, which is similar to that used by BID33, is to assign a complexity weight to each operator (see TAB0), with higher weights assigned to operators considered more complex. If the weight of operator o is c o, then the complexity of an expression tree beginning at node o is defined recursively as DISPLAYFORM0 where o has k arguments, and C(a) is the complexity of argument a. The complexity of a representation is then defined as the sum of the complexities of its output nodes. The goal of defining complexity in such a way is to discourage deep sub-expressions within complex nodes, which are often hard to interpret. It's important to note that the choice of operator weights is bound to be subjective, since we lack an objective notion of interpretability. For this reason, although we use Eqn. 4 to drive search, our experimental comparisons with other algorithms rely on the node counts of the final models for benchmarking interpretability of different methods. Disentanglement is a term used to describe the notion of a representation's ability to separate factors of variation in the underlying process BID3. Although a thorough review is beyond the scope of this section, there is a growing body of literature addressing disentanglement, primarily with unsupervised learning and/or image analysis BID46 BID67 BID21 BID17 BID20 BID36. There are various ways to quantify disentanglement. For instance, Brahma et al. FORMULA2 proposed measuring disentanglement as the difference between geodesic and Euclidean distances among points on a manifold (i.e. training instances). If the latent structure is known, the information-theoretic metrics proposed by BID13 may be used. In the case of regression, a disentangled representation ideally contains a minimal set of features, each corresponding to a separate latent factor of variation, and each orthogonal to each other. In this regard, we attempt to minimize the collinearity between features in φ as a way to promote disentanglement. We tested two measurements of collinearity (a.k.a. multicollinearity) in the derived feature space. The first is the average squared Pearson's correlation among features of φ, i.e., DISPLAYFORM1 Eqn. 5 is relatively inexpensive to compute but only captures bivariate correlations in φ. As a we also test the condition number (CN). Consider the N × m representation matrix Φ. The CN of Φ is defined as DISPLAYFORM2 where µ max and µ min are the largest and smallest singular values of Φ. Unlike Corr, CN can capture higher-order dependencies in the representation. CN is also related directly to the sensitivity of Φ to perturbations in the training data BID2 BID8, and thus captures a notion of network invariance explored in previous work by BID18. We consider another common measure of multicollinearity, the variance inflation factor , to be too expensive for our purposes. The idea to evolve NN architectures is well established in literature, and is known as neuroevolution. Popular methods of neuroevolution include neuroevolution of augmenting topologies (NEAT BID60 and Hyper-NEAT BID61, and compositional pattern producing networks BID59 . The aforementioned approaches eschew the parameter learning step common in other NN paradigms, although others have developed integrations BID14 . In addition, they have been developed predominantly for other task domains such as robotics and control BID16, image classification BID55 BID54, and reinforcement learning BID25 BID9 . Reviews of these methods are available BID15 BID62 .Most neuroevolution strategies do not have interpretability as a core focus, and thus do not attempt to use multi-objective methods to update the networks. An exception is the work of BID68, in which a template NN was optimized using a multi-objective EC method with size as an objective. In this case, the goal was to reduce computational complexity in face detection. Neuroevolution is a part of a broader research field of neural architecture search (NAS) BID70 BID42 BID43. NAS methods vary in approach, including for example parameter sharing BID53, sequential model-based optimization BID43, reinforcement learning BID70, and greedy heuristic strategies BID10.FEAT is also related to SR approaches to feature engineering BID35 BID0 BID41 BID47 ) that use EC to search for possible representations and couple with an ML model to handle the parametrization of the representations. SR methods have been successful in developing intelligible models of physical systems BID57 BID38. FEAT differs from these methods in the following ways. A key challenge in SR is understanding functional modularity within representations/programs that can be exploited for search. FEAT is designed with the insight that ML weights can be leveraged during variation to promote functional building blocks, an exploit not used in previous methods. Second, FEAT uses multiple type representations, and thus can learn continuous and rule-based features within a single representation, unlike previous methods. This is made possible using a stack-based encoding with strongly-typed operators. Finally, FEAT incorporates two elements of NN learning to improve its representational capacity: activation functions commonly used in NN and edge-based encoding of weights. Traditionally, SR operates with standard mathematical operators, and treats constants as leaves in the expression trees rather than edge weights. An exception is MRGP BID0, which encodes weights at each node but updates them via Lasso instead of using gradient descent with backpropagation. SR methods have also been paired with various parameter learning strategies, including those based on backpropagation BID64 BID32 BID27. It should be noted that non-stochastic methods for SR exist, such as mixed integer non-linear programming, which has been demonstrated for small search spaces BID1. Our goals with the experiment are to 1) robustly compare FEAT to state-of-the-art regression methods, including hyperparameter optimization of feedforward NNs; 2) characterize the complexity of the models; and 3) assess whether disentanglement objectives lead to less correlated representations. For the regression datasets, we use 100 real-world and simulated datasets available from OpenML (BID65 . The datasets are characterized in terms of number of features and sample sizes in FIG3 . We use the standardized versions of the datasets available in the Penn Machine Learning Benchmark repository BID48 . We compare the FEAT configurations to multi-layer perceptron (MLP), random forest (RF) regression, kernel ridge (KernelRidge) regression, and elastic net (ElasticNet) regression, using implementations from scikit-learn BID52. In addition, we compare to XGBoost (XGB), a gradient boosting method that has performed well in recent competitions BID6. Code to reproduce these experiments is available online. Each method's hyperparameters are tuned according to TAB1 in Appendix A.1. For FEAT, we limit optimization to 200 iterations or 60 minutes, whichever comes first. We also stop sooner if the median validation fitness stops improving. For each method, we use grid search to tune the hyperparameters with 10-fold cross validation (CV). We use the R 2 CV score for assessing performance. In our we report the CV scores for each method using its best hyperparameters. The algorithms are ranked on each dataset using their median CV score over 5 randomized shuffles of the dataset. For comparing complexity, we count the number of nodes in the final model produced by each method for each trial on each dataset. To quantify the "entanglement" of the feature spaces, we report Eqn. 5 in the raw data and in the final hidden layer of FEAT and MLP models. We also test two additional versions of Feat, denoted FeatCorr and FeatCN, that include a third objective corresponding to Eqn. 5 and 6, respectively. Finally, we examine the FEAT in detail for one of the benchmark datasets. For this dataset we plot the final population of models, illustrate model selection and compare the ant features to from linear and ensemble tree-based . This gives practical insight into the method and provides a sense of the intelligibility of an example representation. The score statistics for each method are shown in Fig. 3 across all datasets. Full statistical comparisons are reported in Appendix A.3. Over all, FEAT and XGBoost produce the best predictive performance across datasets without significant differences between the two (p=1.0). FEAT significantly outperforms MLP, RF, KernelRidge and ElasticNet (p ≤1.18e-4), as does XGBoost (p ≤1.6e-3).As measured by the number of nodes in the final solutions, the models produced by FEAT are significantly less complex than XGBoost, RF, and MLP, as shown in FIG0 ). FEAT's final models tend to be within 1 order of magnitude of the linear models (ElasticNet), and 2-4 orders of magnitude smaller than the other non-linear methods. A comparison of wall-clock time is given in FIG5 in the appendix. FEAT and MLP take approximately the same time to run, followed by XGBoost, RF, KernelRidge, and ElasticNet, in that order. FIG4 shows the average pairwise correlations of the representations produced by Feat variants and MLP in comparison to the correlation structure of the original data. In general, MLP and FEAT tend to produce correlated feature spaces, and Feat's representations tend to contain more bivariate correlations than MLP. Furthermore, the suggest that explicitly minimizing collinearity DISPLAYFORM0 64.8 Figure 6: (Left) Representation archive for the visualizing galaxies dataset. (Right) Selected model and its weights. Internal weights omitted.(FeatCorr and FeatCN) tends to produce representations that exhibit equivalent or higher levels of correlation. This conflicts with our hypothesis, and is discussed more in Section 6. This paper proposes a feature engineering archive tool that optimizes neural network architectures by representing them as syntax trees. FEAT uses model weights as feedback to guide network variation in an EC optimization algorithm. We conduct a thorough analysis of this method applied to the task of regression in comparison to state-of-the-art methods. The suggest that FEAT achieves state-of-the-art performance on regression tasks while producing representations that are significantly less complex than those ing from similarly performing methods. This improvement comes at an additional computational cost, limited in this study to 60 minutes per training instance. We expect this limitation to be reasonable for many applications where intelligibility is the prime motivation. Future work should consider the issue of representation disentanglement in more depth. Our attempts to include additional search objectives that explicitly minimize multicollinearity were not successful. Although more analysis is needed to confirm this, we suspect that the model selection procedure (Section 2.1, step 3) permits highly collinear representations to be chosen. This is because multicollinearity primarily affects the standard errors ofβ BID2, and is not necessarily detrimental to validation error. Therefore it could be incorrect to expect the model selection procedure to effectively choose more disentangled representations. Besides improving the model selection procedure, it may be fruitful to pressure disentanglement at other stages of the search process. For example, the variation process could prune highly correlated features, or the disentanglement metric could be combined with error into a single loss function with a tunable parameter. We hope to pursue these ideas in future studies. This work was supported by NIH grants AI116794 and LM012601.A APPENDIX A.1 ADDITIONAL EXPERIMENT INFORMATION TAB1 details the hyperparameters for each method used in the experimental described in Sections 4 and 5.Runs are conducted in a heterogenous computing environment, with one core assigned to each CV training per dataset. As such, wall-clock times are a flawed measure of computational complexity. With this caveat in mind, we report the wall-clock run times for each method in FIG5. The Feat variants are terminated at 200 generations or 60 minutes, which explains their uniformity. Note that methods are unable to take advantage of parallelization in this experiment. Our initial analysis sought to determine how different SO approaches performed within this framework. We tested five methods: 1) NSGA2, 2) Lex, 3) LexNSGA2, 4) Simulated annealing, and 5) random search. The simulated annealing and random search approaches are described below. Simulated annealing Simulated annealing (SimAnn) is a non-evolutionary technique that instead models the optimization process on the metallurgical process of annealing. In our implementation, offspring compete with their parents; in the case of multiple parents, offspring compete with the program with which they share more nodes. The probability of an offspring replacing its parent in the population is given by the equation DISPLAYFORM0 The probability of offspring replacing its parent is a function of its fitness, F, in our case the mean squared loss of the candidate model. In Eqn. 7, t is a scheduling parameter that controls the rate of "cooling", i.e. the rate at which steps in the search space that are worse are tolerated by the update rule. In accordance with BID31, we use an exponential schedule for t, defined as t g = (0.9) g t 0, where g is the current generation and t0 is the starting temperature. t0 is set to 10 in our experiments. Random search We compare the selection and survival methods to random search, in which no assumptions are made about the structure of the search space. To conduct random search, we randomly sample S using the initialization procedure. Since FEAT begins with a linear model of the process, random search will produce a representation at least as good as this initial model on the internal validation set. A note on archiving When FEAT is used without a complexity-aware survival method (i.e., with Lex, SimAnn, Random), a separate population is maintained that acts as an archive. The archive maintains a Pareto front according to minimum loss and complexity (Eqn 4). At the end of optimization, the archive is tested on a small hold-out validation set. The individual with the lowest validation loss is the final selected model. Maintaining this archive helps protect against overfitting ing from overly complex / high capacity representations, and also can be interpreted directly to help understand the process being modelled. We benchmarked these approaches in a separate experiment on 88 datasets from PMLB BID48. The are shown in FIG0. FIG6, we see that LexNSGA2 achieves the best average R 2 value while producing small solutions in comparison to Lex. NSGA2, SimAnneal, and Random search all produce less accurate models. The runtime comparisons of the methods in FIG0 show that they are mostly within an order of magnitude, with NSGA2 being the fastest (due to its maintenance of small representations) and Random search being the slowest, suggesting that it maintains large representations during search. The computational behavior of Random search suggests the variation operators tend to increase the average size of solutions over many iterations. We perform pairwise comparisons of methods according to the procedure recommended by BID12 for comparing multiple estimators. In TAB3, the CV R 2 rankings are compared. In 1.00e+00 KernelRidge 1.16e-09* 1.18e-04* 4.37e-03* 1.14e-03* MLP 5.24e-09* 3.80e-04* 2.08e-02* 1.28e-03* 1.00e+00 RF 1.08e-09* 2.09e-07* 2.19e-05* 1.30e-06* 1.00e+00 1.00e+00 XGB 1.47e-13* 1.00e+001.00e+00 1.00e+00 3.41e-04* 1.60e-03* 8.49e-13* 1.58e-07* MLP 6.24e-18* 4.26e-17* 3.09e-17* 3.98e-17* RF 9.28e-20* 2.05e-17* 5.61e-18* 2.37e-17* 3.54e-17* XGB 9.14e-18* 4.05e-17* 2.46e-17* 3.71e-17* 1.00e+00 3.94e-18*

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Representing the network architecture as a set of syntax trees and optimizing their structure leads to accurate and concise regression models.

We show shortcut connections should be placed in patterns that minimize between-layer distances during backpropagation, and design networks that achieve log L distances using L log(L) connections.


Question: Represent the scientific article to find a short TLDR

Question: Most distributed machine learning (ML) systems store a copy of the model parameters locally on each machine to minimize network communication. In practice, in order to reduce synchronization waiting time, these copies of the model are not necessarily updated in lock-step, and can become stale. Despite much development in large-scale ML, the effect of staleness on the learning efficiency is inconclusive, mainly because it is challenging to control or monitor the staleness in complex distributed environments. In this work, we study the convergence behaviors of a wide array of ML models and algorithms under delayed updates. Our extensive experiments reveal the rich diversity of the effects of staleness on the convergence of ML algorithms and offer insights into seemingly contradictory reports in the literature. The empirical findings also inspire a new convergence analysis of SGD in non-convex optimization under staleness, matching the best-known convergence rate of O(1/\sqrt{T}). With the advent of big data and complex models, there is a growing body of works on scaling machine learning under synchronous and non-synchronous 1 distributed execution BID8 BID11 BID29. These works, however, point to seemingly contradictory on whether non-synchronous execution outperforms synchronous counterparts in terms of absolute convergence, which is measured by the wall clock time to reach the desired model quality. For deep neural networks, BID2; BID8 show that fully asynchronous systems achieve high scalability and model quality, but others argue that synchronous training converges faster BID1 BID5. The disagreement goes beyond deep learning models:; BID49; BID26; BID31; BID41 empirically and theoretically show that many algorithms scale effectively under non-synchronous settings, but BID36;; demonstrate significant penalties from asynchrony. The crux of the disagreement lies in the trade-off between two factors contributing to the absolute convergence: statistical efficiency and system throughput. Statistical efficiency measures convergence per algorithmic step (e.g., a mini-batch), while system throughput captures the performance of the underlying implementation and hardware. Non-synchronous execution can improve system throughput due to lower synchronization overheads, which is well understood BID1 BID4 BID2. However, by allowing various workers to use stale versions of the model that do not always reflect the latest updates, non-synchronous systems can exhibit lower statistical efficiency BID1 BID5. How statistical efficiency and system throughput trade off in distributed systems, however, is far from clear. The difficulties in understanding the trade-off arise because statistical efficiency and system throughput are coupled during execution in distributed environments. Non-synchronous executions are in general non-deterministic, which can be difficult to profile. Furthermore, large-scale experiments 2 RELATED WORK Staleness is reported to help absolute convergence for distributed deep learning in BID2; BID8; and has minimal impact on convergence BID31 BID6 BID51 BID32. But BID1; BID5 show significant negative effects of staleness. LDA training is generally insensitive to staleness BID44 BID47 BID7, and so is MF training BID48 BID33 BID4 BID49. However, none of their evaluations quantifies the level of staleness in the systems. By explicitly controlling the staleness, we decouple the distributed execution, which is hard to control, from ML convergence outcomes. We focus on algorithms that are commonly used in large-scale optimization BID11 BID1 BID8, instead of methods specifically designed to minimize synchronization BID39 BID43 BID20. Non-synchronous execution has theoretical underpinning BID30 BID49 BID31 BID41. Here we study algorithms that do not necessarily satisfy assumptions in their analyses. We study six ML models and focus on algorithms that lend itself to data parallelism, which a primary approach for distributed ML. Our algorithms span optimization, sampling, and black box variational inference. TAB1 summarizes the studied models and algorithms. Simulation Model. Each update generated by worker p needs to be propagated to both worker p's model cache and other worker's model cache. We apply a uniformly random delay model to these updates that are in transit. Specifically, let u each worker p (including p itself), our delay model applies a delay r BID40 BID9 BID23 BID16 BID12, and dataset BID24 BID34 BID27 BID14 BID42 in our study. η denotes learning rate, which, if not specified, are tuned empirically for each algorithm and staleness level, β1, β2 are optimization hyperparameters (using common default values). α, β in LDA are Dirichlet priors for document topic and word topic random variables, respectively. Convolutional Neural Networks (CNNs) have been a strong focus of large-scale training, both under synchronous BID11 BID5 BID3 and non-synchronous BID2 BID8 BID1 training. We consider residual networks with 6n + 2 weight layers BID15. The networks consist of 3 groups of n residual blocks, with 16, 32, and 64 feature maps in each group, respectively, followed by a global pooling layer and a softmax layer. The residual blocks have the same construction as in BID15. We measure the model quality using test accuracy. For simplicity, we omit data augmentation in our experiments. Deep Neural Networks (DNNs) are neural networks composed of fully connected layers. Our DNNs have 1 to 6 hidden layers, with 256 neurons in each layer, followed by a softmax layer. We use rectified linear units (ReLU) for nonlinearity after each hidden layer BID38. Multiclass Logistic Regression (MLR) is the special case of DNN with 0 hidden layers. We measure the model quality using test accuracy. Matrix factorization (MF) is commonly used in recommender systems and have been implemented at scale BID48 BID33 BID4 BID49 BID22. Let D ∈ R M ×N be a partially filled matrix, MF factorizes D into two factor matrices L ∈ R M ×r and R ∈ R N ×r (r min(M, N) is the user-defined rank). The 2 -penalized optimization problem is: DISPLAYFORM0 where || · || F is the Frobenius norm and λ is the regularization parameter. We partition observations D to workers while treating L, R as shared model parameters. We optimize MF via SGD, and measure model quality by training loss defined by the objective function above. Latent Dirichlet Allocation (LDA) is an unsupervised method to uncover hidden semantics ("topics") from a group of documents, each represented as a bag of tokens. LDA has been scaled under non-synchronous execution BID0 BID33 BID47 with great success. Further details are provided in Appendix. Variational Autoencoder (VAE) is commonly optimized by black box variational inference, which can be considered as a hybrid of optimization and sampling methods. The inputs to VAE training include two sources of stochasticity: the data sampling x and samples of random variable. We measure the model quality by test loss. We use DNNs with 1∼3 layers as the encoders and decoders in VAE, in which each layer has 256 units furnished with rectified linear function for non-linearity. The model quality is measured by the training objective value, assuming continuous input x and isotropic Gaussian prior p(z) ∼ N (0, I). We use batch size 32 for CNNs, DNNs, MLR, and VAEs 34. For MF, we use batch size of 25000 samples, which is 2.5% of the MovieLens dataset (1M samples). We study staleness up to s = 50 on 8 workers, which means model caches can miss updates up to 8.75 data passes. For LDA we use DISPLAYFORM0 as the batch size, where D is the number of documents and P is the number of workers. We study staleness up to s = 20, which means model caches can miss updates up to 2 data passes. We measure time in terms of the amount of work performed, such as the number of batches processed. Convergence Slowdown. Perhaps the most prominent effect of staleness on ML algorithms is the slowdown in convergence, evident throughout the experiments. FIG1 shows the number of batches needed to reach the desired model quality for CNNs and DNNs/MLR with varying network depths and different staleness (s = 0, ..., 16). FIG1 (d) show that convergence under higher level of staleness requires more batches to be processed in order to reach the same model quality. This additional work can potentially be quite substantial, such as in FIG1 where it takes up to 6x more batches compared with settings without staleness (s = 0). It is also worth pointing out that while there can be a substantial slowdown in convergence, the optimization still reaches desirable models under most cases in our experiments. When staleness is geometrically distributed FIG3 ), we observe similar patterns of convergence slowdown. We are not aware of any prior work reporting slowdown as high as observed here. This finding has important ramifications for distributed ML. Usually, the moderate amount of workload increases due to parallelization errors can be compensated by the additional computation resources and higher system throughput in the distributed execution. However, it may be difficult to justify spending large 3 Non-synchronous execution allows us to use small batch sizes, eschewing the potential generalization problem with large batch SGD BID21 BID35. 4 We present RNN in the Appendix.amount of resources for a distributed implementation if the statistical penalty is too high, which should be avoided (e.g., by staleness minimization system designs or synchronous execution).Model Complexity. FIG1 also reveals that the impact of staleness can depend on ML parameters, such as the depths of the networks. Overall we observe that staleness impacts deeper networks more than shallower ones. This holds true for SGD, Adam, Momentum, RMSProp, Adagrad FIG1, and other optimization schemes, and generalizes to other numbers of workers (see Appendix) 5.This is perhaps not surprising, given the fact that deeper models pose more optimization challenges even under the sequential settings BID10 BID15, though we point out that existing literature does not explicitly consider model complexity as a factor in distributed ML BID31 BID11. Our suggest that the staleness level acceptable in distributed training can depend strongly on the complexity of the model. For sufficiently complex models it may be more advantageous to eliminate staleness altogether and use synchronous training. Algorithms' Sensitivity to Staleness. Staleness has uneven impacts on different SGD variants. Fig. 2 shows the amount of work (measured in the number of batches) to reach the desired model quality for five SGD variants. Fig. 2 (d)(e)(f) reveals that while staleness generally increases the number of batches needed to reach the target test accuracy, the increase can be higher for certain algorithms, such as Momentum. On the other hand, Adagrad appear to be robust to staleness 6. Our finding is consistent with the fact that, to our knowledge, all existing successful cases applying non-synchronous training to deep neural networks use SGD BID8 BID2. In contrast, works reporting subpar performance from non-synchronous training often use momentum, such as RMSProp with momentum BID1 and momentum BID5. Our suggest that these different outcomes may be partly driven by the choice of optimization algorithms, leading to the seemingly contradictory reports of whether non-synchronous execution is advantageous over synchronous ones. Effects of More Workers. The impact of staleness is amplified by the number of workers. In the case of MF, Fig. 3(b) shows that the convergence slowdown in terms of the number of batches (normalized by the convergence for s = 0) on 8 workers is more than twice of the slowdown on 4 workers. For example, in Fig. 3 (b) the slowdown at s = 15 is ∼3.4, but the slowdown at the same staleness level on 8 workers is ∼8.2. Similar observations can be made for CNNs (Fig. 3). This can be explained by the fact that additional workers amplifies the effect of staleness by generating updates that will be subject to delays, and missing updates from other workers that are subject to delays. Fig. 3 (c)(d) show the convergence curves of LDA with different staleness levels for two settings varying on the number of workers and topics. Unlike the convergence curves for SGD-based algorithms (see Appendix), the convergence curves of Gibbs sampling are highly smooth, even under high staleness and a large number of workers. This can be attributed to the structure of log likelihood objective function BID12. Since in each sampling step we only update the count statistics based on a portion of the corpus, the objective value will generally change smoothly. Staleness levels under a certain threshold (s ≤ 10) lead to convergence, following indistinguishable log likelihood trajectories, regardless of the number of topics (K = 10, 100) or the number of workers (2-16 workers, see Appendix). Also, there is very minimal variance in those trajectories. However, for staleness beyond a certain level (s ≥ 15), Gibbs sampling does not converge to a fixed point. The convergence trajectories are distinct and are sensitive to the number of topics and the number of workers. There appears to be a "phase transition" at a certain staleness level that creates two distinct phases of convergence behaviors 7. We believe this is the first report of a staleness-induced failure case for LDA Gibbs sampling. Fig. 3 (e)(f), VAEs exhibit a much higher sensitivity to staleness compared with DNNs (FIG1). This is the case even considering that VAE with depth 3 has 6 weight layers, which has a comparable number of model parameters and network architecture to DNNs with 6 layers. We hypothesize that this is caused by the additional source of stochasticity from the sampling procedure, in addition to the data sampling process. We now provide theoretical insight into the effect of staleness on the observed convergence slowdown. We focus on the challenging asynchronous SGD (Async-SGD) case, which characterizes the neural network models, among others. Consider the following nonconvex optimization problem DISPLAYFORM0 where f i corresponds to the loss on the i-th data sample, and the objective function is assumed to satisfy the following standard conditions:Assumption 1. The objective function F in the problem (P) satisfies:1. Function F is continuously differentiable and bounded below, i.e., inf x∈R d F (x) > −∞; 2. The gradient of F is L-Lipschitz continuous. Notice that we allow F to be nonconvex. We apply the Async-SGD to solve the problem (P). Let ξ(k) be the mini-batch of data indices sampled from {1, . . ., n} uniformly at random by the algorithm at iteration k, and |ξ(k)| is the mini-batch size. Denote mini-batch gradient as ∇f ξ(k) (x k):= i∈ξ(k) ∇f i (x k). Then, the update rule of Async-SGD can be written as DISPLAYFORM1 where η k corresponds to the stepsize, τ k denotes the delayed clock and the maximum staleness is assumed to be bounded by s. This implies that DISPLAYFORM2 The optimization dynamics of Async-SGD is complex due to the nonconvexity and the uncertainty of the delayed updates. Interestingly, we find that the following notion of gradient coherence provides insights toward understanding the convergence property of Async-SGD.Definition 1 (Gradient coherence). The gradient coherence at iteration k is defined as DISPLAYFORM3 Parameter µ k captures the minimum coherence between the current gradient ∇F (x k) and the gradients along the past s iterations 8. Intuitively, if µ k is positive, then the direction of the current gradient is well aligned to those of the past gradients. In this case, the convergence property induced by using delayed stochastic gradients is close to that induced by using synchronous stochastic gradients. Note that Definition 1 only requires the gradients to be positively correlated over a small number of iterations s, which is often very small (e.g. <10 in our experiments). Therefore, Definition 1 is not a global requirement on optimization path. Even though neural network's loss function is non-convex, recent studies showed strong evidences that SGD in practical neural network training encourage positive gradient coherence BID28 BID32. This is consistent with the findings that the loss surface of shallow networks and deep networks with skip connections are dominated by large, flat, nearly convex attractors around the critical points BID28 BID21, implying that the degree of non-convexity is mild around critical points. We show in the sequel that µ k > 0 through most of the optimization path, especially when the staleness is minimized in practice by system optimization FIG3. Our theory can be readily adapted to account for a limited amount of negative µ k (see Appendix), but our primary interest is to provide a quantity that is easy to compute empirically during the course of optimization 9, and informative for the impact of staleness and can potentially be used to control synchronization levels. We now characterize the convergence property of Async-SGD. Theorem 1. Let Assumption 1 hold. Suppose for some µ > 0, the gradient coherence satisfies µ k ≥ µ for all k and the variance of the stochastic gradients is bounded by σ 2 > 0. Choose stepsize DISPLAYFORM4. Then, the iterates generated by the Async-SGD satisfy DISPLAYFORM5 8 Our gradient coherence bears similarity with the sufficient direction assumption in BID19. However, sufficient direction is a layer-wise and fixed delay, whereas our staleness is a random variable that is subject to system level factors such as communication bandwidth 9 It can be approximated by storing a pre-selected batch of data on a worker. The worker just needs to compute gradient every T mini-batches to obtain approximate ∇F (x k), ∇F (xt) in Definition 1. TAB1. Shaded region is 1 standard deviation over 3 runs. For computational efficiency, we approximate the full gradient ∇F (x k) by gradients on a fixed set of 1000 training samples D f ixed and use ∇D f ixed F (x k). (c) The number of batches to reach 71% test accuracy on CIFAR10 for ResNet8-32 using 8 workers and SGD under geometric delay distribution (details in Appendix).We refer readers to Appendix for the the proof. Theorem 1 characterizes several theoretical aspects of Async-SGD. First, the choice of the stepsize η k = µ sL √ k is adapted to both the maximum staleness and the gradient coherence. Intuitively, if the system encounters a larger staleness, then a smaller stepsize should be used to compensate the negative effect. On the other hand, the stepsize can be accordingly enlarged if the gradient coherence along the iterates turns out to be high. In this case, the direction of the gradient barely changes along the past several iterations, and a more aggressive stepsize can be adopted. In summary, the choice of stepsize should trade-off between the effects caused by both the staleness and the gradient coherence. depths optimized by SGD using 8 workers. The x-axis m is defined in FIG3 Furthermore, Theorem 1 shows that the minimum gradient norm decays at the rate O(DISPLAYFORM6), implying that the Async-SGD converges to a stationary point provided a positive gradient coherence, which we observe empirically in the sequel. On the other hand, the bound in Eq. captures the trade-off between the maximum staleness s and the gradient coherence µ. Specifically, minimizing the right hand side of Eq. with regard to the maximum staleness s yields the optimal choice s * = σµ log T L(F (x0)−infx F (x)), i.e., a larger staleness is allowed if the gradients remain to be highly coherent along the past iterates. Empirical Observations. Theorem 1 suggests that more coherent gradients along the optimization paths can be advantageous under non-synchronous execution. FIG3 shows the cosine similarity sim(a, b):= a·b a b between gradients along the convergence path for CNNs and DNNs 10. We observe the followings: Cosine similarity improves over the course of convergence FIG3 ). Except the highest staleness during the early phase of convergence, cosine similarity remains positive 11. In practice the staleness experienced during run time can be limited to small staleness, which minimizes the likelihood of negative gradient coherence during the early phase. FIG4 shows that cosine similarity decreases with increasing CNN model complexity. Theorem 1 implies that lower gradient coherence amplifies the effect of staleness s through the factor s µ 2 in Eq.. This is consistent with the convergence difficulty encountered in deeper models FIG1. In this work, we study the convergence behaviors under delayed updates for a wide array of models and algorithms. Our extensive experiments reveal that staleness appears to be a key governing parameter in learning. Overall staleness slows down the convergence, and under high staleness levels the convergence can progress very slowly or fail. The effects of staleness are highly problem 10 Cosine similarity is closely related to the coherence measure in Definition 1. 11 Low gradient coherence during the early part of optimization is consistent with the common heuristics to use fewer workers at the beginning in asynchronous training. BID31 also requires the number of workers to follow DISPLAYFORM0 where K is the iteration number.dependent, influenced by model complexity, choice of the algorithms, the number of workers, and the model itself, among others. Our empirical findings inspire new analyses of non-convex optimization under asynchrony based on gradient coherence, matching the existing rate of O(1/ √ T).Our findings have clear implications for distributed ML. To achieve actual speed-up in absolute convergence, any distributed ML system needs to overcome the slowdown from staleness, and carefully trade off between system throughput gains and statistical penalties. Many ML methods indeed demonstrate certain robustness against low staleness, which should offer opportunities for system optimization. Our support the broader observation that existing successful nonsynchronous systems generally keep staleness low and use algorithms efficient under staleness. A.1 PROOF OF THEOREM 1Theorem 2. Let Assumption 1 hold. Suppose the gradient coherence µ k is lower bounded by some µ > 0 for all k and the variance of the stochastic gradients is upper bounded by some σ 2 > 0. DISPLAYFORM0. Then, the iterates generated by the Async-SGD satisfy DISPLAYFORM1 Proof. By the L-Lipschitz property of ∇F, we obtain that for all k DISPLAYFORM2 Taking expectation on both sides of the above inequality and note that the variance of the stochastic gradient is bounded by σ 2, we further obtain that DISPLAYFORM3 Telescoping the above inequality over k from 0 to T yields that DISPLAYFORM4 Rearranging the above inequality and note that DISPLAYFORM5 Note that the choice of stepsize guarantees that η k µ − > 0 for all k. Thus, we conclude that DISPLAYFORM6 where the last inequality uses the fact that DISPLAYFORM7 into the above inequality and simplifying, we finally obtain that DISPLAYFORM8 A.2 HANDLING NEGATIVE GRADIENT COHERENCE IN THEOREM 1Our assumption of positive gradient coherence (GC) is motivated by strong empirical evidence that GC is largely positive FIG3 in the main text). Contrary to conventional wisdom, GC generally improves when approaching convergence for both SGD and Adam. Furthermore, in practice, the effective staleness for any given iteration generally concentrates in low staleness for the non-stragglers.When some µ k are negative at some iterations, in eq. 11 in the Appendix we can move the negative terms in k η k µ k to the right hand side and yield a higher upper bound (i.e., slower convergence). This is also consistent with empirical observations that higher staleness lowers GC and slows convergence. A.3 EXPONENTIAL DELAY DISTRIBUTION.We consider delays drawn from geometric distribution (GD), which is the discrete version of exponential distribution. For each iterate we randomly select a worker to be the straggler with large mean delay (p = 0.1), while all other non-straggler workers have small delays. The non-straggler delay is drawn from GD with p chosen to achieve the same mean delay as in the uniform case (after factoring in straggler) in the main text. The delay is drawn per worker for each iteration, and thus a straggler's outgoing updates to all workers suffer the same delay. FIG3 (c) in the main text shows the convergence speed under the corresponding staleness s with the same mean delay (though s is not a parameter in GD). It exhibits trends analogous to FIG1 in the main text: staleness slows convergence substantially and overall impacts deeper networks more. We present additional for DNNs. Fig. 6 shows the number of batches, normalized by s = 0, to reach convergence using 1 hidden layer and 1 worker under varying staleness levels and batch sizes. Overall, the effect of batch size is relatively small except in high staleness regime (s = 32). Fig. 7 shows the number of batches to reach convergence, normalized by s = 0 case, for 5 variants of SGD using 1 worker. The are in line with the analyses in the main text: staleness generally leads to larger slow down for deeper networks than shallower ones. SGD and Adagrad are more robust to staleness than Adam, RMSProp, and SGD with momentum. In particular, RMSProp exhibit high variance in batches to convergence (not shown in the normalized plot) and thus does not exhibit consistent trend. The are consistent with the observations and analyses in the main text, namely, that having more workers amplifies the effect of staleness. We can also observe that SGDS is more robust to staleness than Adam, and shallower networks are less impacted by staleness. In particular, note that staleness sometimes accelerates convergence, such as in FIG6. This is due to the implicit momentum created by staleness.A.5 LDA AND ADDITIONAL FOR LDAIn LDA each token w ij (j-th token in the i-th document) is assigned with a latent topic z ij from totally K topics. We use Gibbs sampling to infer the topic assignments z ij. The Gibbs sampling step involves three sets of parameters, known as sufficient statistics: FORMULA7 Figure 6: The number of batches to reach 95% test accuracy using 1 hidden layer and 1 worker, respectively normalized by s = 0.vector φ w ∈ R K where φ wk is the number of topic assignments to topic k = 1,..., K for word (vocabulary) w across all documents;φ ∈ R K whereφ k = W w=1 φ wk is the number of tokens in the corpus assigned to topic k. The corpus (w ij, z ij) is partitioned to workers, while φ w andφ are shared model parameters. We measure the model quality using log likelihood. We present additional of LDA under different numbers of workers and topics in FIG12 and FIG1. These panels extends Fig. 3(c)(d) in the main text. See the main text for experimental setup and analyses and experimental setup. We show the convergence curves for MF under different numbers of workers and staleness levels in FIG1. It is evident that higher staleness leads to a higher variance in convergence. Furthermore, the number of workers also affects variance, given the same staleness level. For example, MF with 4 workers incurs very low standard deviation up to staleness 20. In contrast, MF with 8 workers already exhibits a large variance at staleness 15. The amplification of staleness from increasing number of Figure 7: The number of batches to reach 92% test accuracy using DNNs with varying numbers of hidden layers under 1 worker. We consider several variants of SGD algorithms (a)-(e). Note that with depth 0 the model reduces to MLR, which is convex. The numbers are averaged over 5 randomized runs. We omit the whenever convergence is not achieved within the experiment horizon (77824 batches), such as SGD with momentum at depth 6 and s = 32. Recurrent Neural Networks (RNNs) are widely used in recent natural language processing tasks. We consider long short-term memory (LSTM) BID18 applied to the language modeling task, using a subset of Penn Treebank dataset (PTB) BID34 containing 5855 words. The dataset is pre-processed by standard de-capitalization and tokenization. We evaluate the impact of staleness for LSTM with 1 to 4 layers, with 256 neurons in each layer. The maximum length for each sentence is 25. Note that 4 layer LSTM is about 4x more model parameters than the 1 layer LSTM, which is the same ratio between ResNet32 and Resnet 8. We use batch size 32 similar to other experiments. We consider staleness s = 0, 4, 8, 16 on 8 workers. The model quality is measured in perplexity. FIG1 shows the number of batches needed to reach the desired model quality for RNNs with varying network depths. We again observe that staleness impacts deeper network variants more than shallower counterparts, which is consistent with our observation in CNNs and DNNs.

represent the tldr for finding the corresponding scientific article.
To give you a sense - "We propose a novel tensor based method for graph convolutional networks on dynamic graphs" should be close to "Many irregular domains such as social networks, financial transactions, neuron connections, and natural language structures are represented as graphs. In recent years, a variety of graph neural networks (GNNs) have been successfully applied for representation learning and prediction on such graphs. However, in many of the applications, the underlying graph changes over time and existing GNNs are inadequate for handling such dynamic graphs. In this paper we propose a novel technique for learning embeddings of dynamic graphs based on a tensor algebra framework. Our method extends the popular graph convolutional network (GCN) for learning representations of dynamic graphs using the recently proposed tensor M-product technique. Theoretical that establish the connection between the proposed tensor approach and spectral convolution of tensors are developed. Numerical experiments on real datasets demonstrate the usefulness of the proposed method for an edge classification task on dynamic graphs. Graphs are popular data structures used to effectively represent interactions and structural relationships between entities in structured data domains. Inspired by the success of deep neural networks for learning representations in the image and language domains, recently, application of neural networks for graph representation learning has attracted much interest. A number of graph neural network (GNN) architectures have been explored in the contemporary literature for a variety of graph related tasks and applications (; ; ;). Methods based on graph convolution filters which extend convolutional neural networks (CNNs) to irregular graph domains are popular (; ;). Most of these GNN models operate on a given, static graph. In many real-world applications, the underlining graph changes over time, and learning representations of such dynamic graphs is essential. Examples include analyzing social networks , predicting collaboration in citation networks , detecting fraud and crime in financial networks , traffic control , and understanding neuronal activities in the brain . In such dynamic settings, the temporal interdependence in the graph connections and features also play a substantial role. However, efficient GNN methods that handle time varying graphs and that capture the temporal correlations are lacking. By dynamic graph, we mean a sequence of graphs (V, A (t), X (t) ), t ∈ {1, 2, . . ., T}, with a fixed set V of N nodes, adjacency matrices A (t) ∈ R N ×N, and graph feature matrices X (t) ∈ R N ×F where X (t) n: ∈ R F is the feature vector consisting of F features associated with node n at time t. The graphs can be weighted, and directed or undirected. They can also have additional properties like (time varying) node and edge classes, which would be stored in a separate structure. Suppose we only observe the first T < T graphs in the sequence. The goal of our method is to use these observations to predict some property of the remaining T − T graphs. In this paper, we use it for edge classification. Other potential applications are node classification and edge/link prediction. In recent years, tensor constructs have been explored to effectively process high-dimensional data, in order to better leverage the multidimensional structure of such data . Tensor based approaches have been shown to perform well in many image and video processing ap- plications;;;; ). A number of tensor based neural networks have also been investigated to extract and learn multi-dimensional representations, e.g. methods based on tensor decomposition , tensor-trains , and tensor factorized neural network . Recently, a new tensor framework called the tensor M-product framework (; ;) was proposed that extends matrix based theory to high-dimensional architectures. In this paper, we propose a novel tensor variant of the popular graph convolutional network (GCN) architecture , which we call TensorGCN. It captures correlation over time by leveraging the tensor M-product framework. The flexibility and matrix mimeticability of the framework, help us adapt the GCN architecture to tensor space. Figure 1 illustrates our method at a high level: First, the time varying adjacency matrices A (t) and feature matrices X (t) of the dynamic graph are aggregated into an adjacency tensor and a feature tensor, respectively. These tensors are then fed into our TensorGCN, which computes an embedding that can be used for a variety of tasks, such as link prediction, and edge and node classification. GCN architectures are motivated by graph convolution filtering, i.e., applying filters/functions to the graph Laplacian (in turn its eigenvalues) , and we establish a similar connection between TensorGCN and spectral filtering of tensors. Experimental on real datasets illustrate the performance of our method for the edge classification task on dynamic graphs. Elements of our method can also be used as a preprocessing step for other dynamic graph methods. The idea of using graph convolution based on the spectral graph theory for GNNs was first introduced by. then proposed Chebnet, where the spectral filter was approximated by Chebyshev polynomials in order to make it faster and localized. presented the simplified GCN, a degree-one polynomial approximation of Chebnet, in order to speed up computation further and improve the performance. There are many other works that deal with GNNs when the graph and features are fixed/static; see the review papers by and and references therein. These methods cannot be directly applied to the dynamic setting we consider. devised the Graph Convolutional Recurrent Network for graphs with time varying features. However, this method assumes that the edges are fixed over time, and is not applicable in our setting. proposed a method called EdgeConv, which is a neural network (NN) approach that applies convolution operations on static graphs in a dynamic fashion. Their approach is not applicable when the graph itself is dynamic. develop a temporal GCN method called T-GCN, which they apply for traffic prediction. Their method assumes the graph remains fixed over time, and only the features vary. The set of methods most relevant to our setting of learning embeddings of dynamic graphs use combinations of GNNs and recurrent architectures (RNN), to capture the graph structure and handle time dynamics, respectively. The approach in uses Long Short-Term Memory (LSTM), a recurrent network, in order to handle time variations along with GNNs. They design architectures for semi-supervised node classification and for supervised graph classification. presented a variant of GCN called EvolveGCN, where Gated Recurrent Units (GRU) and LSTMs are coupled with a GCN to handle dynamic graphs. This paper is currently the stateof-the-art. However, their approach is based on a heuristic RNN/GRU mechanism, which is not theoretically viable, and does not harness a tensor algebraic framework to incorporate time varying information. present a tensor NN which utilizes the M-product tensor framework. Their approach can be applied to image and other high-dimensional data that lie on regular grids, and differs from ours since we consider data on dynamic graphs. Here, we cover the necessary preliminaries on tensors and the M-product framework. For a more general introduction to tensors, we refer the reader to the review paper by. In this paper, a tensor is a three-dimensional array of real numbers denoted by boldface Euler script letters, e.g. X ∈ R I×J×T. Matrices are denoted by bold uppercase letters, e.g. X; vectors are denoted by bold lowercase letter, e.g. x; and scalars are denoted by lowercase letters, e.g. x. An element at position (i, j, t) in a tensor is denotes by subscripts, e.g. X ijt, with similar notation for elements of matrices and vectors. A colon will denote all elements along that dimension; X i: denotes the ith row of the matrix X, and X::k denotes the kth frontal slice of X. The vectors X ij: are called the tubes of X. The framework we consider relies on a new definition of the product of two tensors, called the M-product (; ;). A distinguishing feature of this framework is that the M-product of two three-dimensional tensors is also three-dimensional, which is not the case for e.g. tensor contractions . It allows one to elegantly generalize many classical numerical methods from linear algebra, and has been applied e.g. in neural networks , imaging; ), facial recognition, and tensor completion and denoising (; ;). Although the framework was originally developed for three-dimensional tensors, which is sufficient for our purposes, it has been extended to handle tensors of dimension greater than three . The following definitions 3.1-3.3 describe the M-product. Definition 3.1 (M-transform). Let M ∈ R T ×T be a mixing matrix. The M-transform of a tensor X ∈ R I×J×T is denoted by X × 3 M ∈ R I×J×T and defined elementwise as We say that X × 3 M is in the transformed space. may also be written in matrix form as, where the unfold operation takes the tubes of X and stack them as columns into a T × IJ matrix, and fold(unfold(X)) = X. Appendix A provides illustrations of how the M-transform works. Definition 3.2 (Facewise product). Let X ∈ R I×J×T and Y ∈ R J×K×T be two tensors. The I×J×T and Y ∈ R J×K×T be two tensors, and let M ∈ R T ×T be an invertible matrix. The M-product, denoted by X Y ∈ R I×K×T, is defined as In the original formulation of the M-product, M was chosen to be the Discrete Fourier Transform (DFT) matrix, which allows efficient computation using the Fast Fourier Transform (FFT) (; ; . The framework was later extended for arbitrary invertible M (e.g. discrete cosine and wavelet transforms) . A benefit of the tensor M-product framework is that many standard matrix concepts can be generalized in a straightforward manner. Definitions 3.4-3.7 extend the matrix concepts of diagonality, identity, transpose and orthogonality to tensors (; . Definition 3.5 (Identity tensor). LetÎ ∈ R N ×N ×T be defined facewise asÎ::t = I, where I is the matrix identity. The M-product identity tensor I ∈ R N ×N ×T is then defined as Definition 3.6 (Tensor transpose). The transpose of a tensor X is defined as X def = Y × 3 M −1, where Y::t = (X × 3 M)::t for each t ∈ {1, . . ., T}. Definition 3.7 (Orthogonal tensor). A tensor X ∈ R N ×N ×T is said to be orthogonal if X X = X X = I. Leveraging these concepts, a tensor eigendecomposition can now be defined (; : Definition 3.8 (Tensor eigendecomposition). Let X ∈ R N ×N ×T be a tensor and assume that each frontal slice (X × 3 M)::t is symmetric. We can then eigendecompose these as (X × 3 M)::t = Q::tD::tQ::t, whereQ::t ∈ R N ×N is orthogonal andD::t ∈ R N ×N is diagonal (see e.g. Theorem 8.1.1 in). The tensor eigendecomposition of X is then defined as Our approach is inspired by the first order GCN by for static graphs, owed to its simplicity and effectiveness. For a graph with adjacency matrix A and feature matrix X, a GCN layer takes the form Y = σ(ÃXW), wherẽ is the matrix identity, W is a matrix to be learned when training the NN, and σ is an activation function, e.g., ReLU. Our approach translates this to a tensor model by utilizing the M-product framework. We first introduce a tensor activation functionσ which operates in the transformed space. Definition 4.1. Let A ∈ R I×J×T be a tensor and σ an elementwise activation function. We define the activation functionσ asσ(A) We can now define our proposed dynamic graph embedding. Let A ∈ R N ×N ×T be a tensor with frontal slices A::t =Ã (t), whereÃ (t) is the normalization of A (t). Moreover, let X ∈ R N ×F ×T be a tensor with frontal slices X::t = X (t). Finally, let W ∈ R F ×F ×T be a weight tensor. We define our dynamic graph embedding as Y = A X W ∈ R N ×F ×T. This computation can also be repeated in multiple layers. For example, a 2-layer formulation would be of the form One important consideration is how to choose the matrix M which defines the M-product. For time-varying graphs, we choose M to be lower triangular and banded so that each frontal slice (A × 3 M)::t is a linear combination of the adjacency matrices A::max(1,t−b+1),..., A::t, where we refer to b as the "bandwidth" of M. This choice ensures that each frontal slice (A × 3 M)::t only contains information from current and past graphs that are close temporally. Specifically, the entries of M are set to otherwise, which implies that k M tk = 1 for each t. Another possibility is to treat M as a parameter matrix to be learned from the data. In order to avoid over-parameterization and improve the performance, we choose the weight tensor W (at each layer), such that each of the frontal slices of W in the transformed domain remains the same, i.e., (W × 3 M)::t = (W × 3 M)::t ∀t, t. In other words, the parameters in each layer are shared and learned over all the training instances. This reduces the number of parameters to be learned significantly. An embedding Y ∈ R N ×F ×T can now be used for various prediction tasks, like link prediction, and edge and node classification. In Section 5, we apply our method for edge classification by using a model similar to that used by: Given an edge between nodes m and n at time t, the predictive model is where (Y × 3 M) m:t ∈ R F and (Y × 3 M) n:t ∈ R F are row vectors, U ∈ R C×2F is a weight matrix, and C the number of classes. Note that the embedding Y is first M-transformed before the matrix U is applied to the appropriate feature vectors. This, combined with the fact that the tensor activation functions are applied elementwise in the transformed domain, allow us to avoid ever needing to apply the inverse M-transform. This approach reduces the computational cost, and has been found to improve performance in the edge classification task. Here, we present the that establish the connection between the proposed TensorGCN and spectral convolution of tensors, in particular spectral filtering and approximation on dynamic graphs. This is analogous to the graph convolution based on spectral graph theory in the GNNs by , , and. All proofs are provided in Appendix D. Let L ∈ R N ×N ×T be a form of tensor Laplacian defined as L def = I − A. Throughout the remainder of this subsection, we will assume that the adjacency matrices A (t) are symmetric. Following the work by, three-dimensional tensors in R M ×N ×T can be viewed as operators on N × T matrices, with those matrices "twisted" into tensors in R N ×1×T. With this in mind, we define a tensor variant of the graph Fourier transform. Definition 4.4 (Tensor-tube M-product). Let X ∈ R I×J×T and θ ∈ R 1×1×T. Analogously to the definition of the matrix-scalar product, we define X θ via (X θ) ij: Definition 4.5 (Tensor graph Fourier transform). Let X ∈ R N ×F ×T be a tensor. We define a tensor graph Fourier transform F as F (X) This is analogous to the definition of the matrix graph Fourier transform. This defines a convolution like operation for tensors similar to spectral graph convolution . Each lateral slice X:j: is expressible in terms of the set {Q :n:} N n=1 as follows: where each (Q X :j:) n1: ∈ R 1×1×T can be considered a tubal scalar. In fact, the lateral slices Q:n: form a basis for the set R N ×1×T with product; see Appendix D for further details. Definition 4.6 (Tensor spectral graph filtering). Given a signal X ∈ R N ×1×T and a function g: R 1×1×T → R 1×1×T, we define the tensor spectral graph filtering of X with respect to g as where In order to avoid the computation of an eigendecomposition, use a polynomial to approximate the filter function. We take a similar approach, and approximate g(D) with an M-product polynomial. For this approximation to make sense, we impose additional structure on g. Assumption 4.7. Assume that g: where f is defined elementwise as Proposition 4.8. Suppose g satisfies Assumption 4.7. For any ε > 0, there exists an integer K and a set {θ where · is the tensor Frobenius norm, and where As in the work of , a tensor polynomial approximation allows us to approximate X filt in without computing the eigendecomposition of L: All that is necessary is to compute tensor powers of L. We can also define tensor polynomial analogs of the Chebyshev polynomials and do the approximation in in terms of those instead of the tensor monomials D k. This is not necessary for the purposes of this paper. Instead, we note that if a degree-one approximation is used, the computation in becomes, which is analogous to the parameter choice made in the degree-one approximation by , we get If we let X contain F signals, i.e., X ∈ R N ×F ×T, and apply F filters, becomes where Θ ∈ R F ×F ×T. This is precisely our embedding model, with Θ replaced by a learnable parameter tensor W. Here, we present for edge classification on four datasets 1: The Bitcoin Alpha and OTC transaction datasets , the Reddit body hyperlink dataset , and a chess dataset . The bitcoin datasets consist of transaction histories for users on two different platforms. Each node is a user, and each directed edge indicates a transaction and is labeled with an integer between −10 and 10 which indicates the senders trust for the receiver. We convert these labels to two classes: positive (trustworthy) and negative (untrustworthy). The Reddit dataset is build from hyperlinks from one subreddit to another. Each node represents a subreddit, and each directed edge is an interaction which is labeled with −1 for a hostile interaction or +1 for a friendly interaction. We only consider those subreddits which have a total of 20 interactions or more. In the chess dataset, each node is a player, and each directed edge represents a match with the source node being the white player and the target node being the black player. Each edge is labeled −1 for a black victory, 0 for a draw, and +1 for a white victory. The data is temporally partitioned into T graphs, with each graph containing data from a particular time window. Both T and the time window length can vary between datasets. For each node-time pair (n, t) in these graphs, we compute the number of outgoing and incoming edges and use these two numbers as features. The adjacency tensor A is then constructed as described in Section 4. The T frontal slices of A are divided into S train training slices, S val validation slices, and S test testing slices, which come sequentially after each other; see Figure 2 and Table 2. Since the adjacency matrices corresponding to graphs are very sparse for these datasets, we apply the same technique as and add the entries of each frontal slice A::t to the following l − 1 frontal slices A::t,..., A::(t+l−1), where we refer to l as the "edge life." Note that this only affects A, and that the added edges are not treated as real edges in the classification problem. The bitcoin and Reddit datasets are heavily skewed, with about 90% of edges labeled positively, and the remaining labeled negatively. Since the negative instances are more interesting to identify (e.g. to prevent financial fraud or online hostility), we use the F1 score to evaluate the experiments on these datasets, treating the negative edges as the ones we want to identify. The classes are more well-balanced in the chess dataset, so we use accuracy to evaluate those experiments. We choose to use an embedding Y train = A::(1:Strain) X::(1:Strain) W for training. When computing the embeddings for the validation and testing data, we still need S train frontal slices of A, which we get by using a sliding window of slices. This is illustrated in Figure 2, where the green, blue and red blocks show the frontal slices used when computing the embeddings for the training, validation and testing data, respectively. The embeddings for the validation and testing data are Y val = A::(Sval+1:Strain+Sval) X::(Sval+1:Strain+Sval) W and Y test = A::(Sval+Stest+1:T) X::(Sval+Stest+1:T) W, respectively. Preliminary experiments with 2-layer architectures did not show convincing improvements in performance. We believe this is due to the fact that the datasets only have two features, and that a 1-layer architecture therefore is sufficient for extracting relevant information in the data. For training, we use the cross entropy loss function: where f (m, n, t) ∈ R C is a one-hot vector encoding the true class of the edge (m, n) at time t, and α ∈ R C is a vector summing to 1 which contains the weight of each class. Since the bitcoin and Reddit datasets are so skewed, we weigh the minority class more heavily in the loss function for those datasets, and treat α as a hyperparameter; see Appendix C for details. The experiments are implemented in PyTorch with some preprocessing done in Matlab. Our code is available at [url redacted for review]. In the experiments, we use an edge life of l = 10, a bandwidth b = 20, and F = 6 output features. Since the graphs in the considered datasets are directed, we also investigate the impact of symmetrizing the adjacency matrices, where the symmetrized version of an adjacency matrix A is defined as A sym def = 1/2(A + A). We compare our method with three other methods. The first one is a variant of the WD-GCN by , which they specify in Equation (8a) of their paper. For the LSTM layer in their description, we use 6 output features instead of N. This is to avoid overfitting and make the method more comparable to ours which uses 6 output features. For the final layer, we use the same prediction model as that used by for edge classification. The second method is a 1-layer variant of EvolveGCN-H by. The third method is a simple baseline which uses a 1-layer version of the GCN by. It uses the same weight matrix W for all temporal graphs. Both EvolveGCN-H and the baseline GCN use 6 output features as well. Table 3 shows the when the adjacency matrices have not been symmetrized. In this case, our method outperforms the other methods on the two bitcoin datasets and the chess dataset, with WD-GCN performing best on the Reddit dataset. Table 4 shows the for when the adjacency matrices have been symmetrized. Our method outperforms the other methods on the Bitcoin OTC dataset and the chess dataset, and performs similarly but slightly worse than the best performing methods on the Bitcoin Alpha and Reddit datasets. Overall, it seems like symmetrizing the adjacency matrices leads to lower performance. We have presented a novel approach for dynamic graph embedding which leverages the tensor Mproduct framework. We used it for edge classification in experiments on four real datasets, where it performed competitively compared to state-of-the-art methods. Future research directions include further developing the theoretical guarantees for the method, investigating optimal structure and learning of the transform matrix M, using the method for other prediction tasks, and investigating how to utilize deeper architectures for dynamic graph learning. We provide some illustrations that show how the M-transform in Definition 3.1 works. Recall that X × 3 M = fold(M unfold(X)). The matrix X is first unfolded into a matrix, as illustrated in Figure 3. This unfolded tensor is then multiplied from the left by the matrix M, as illustrated in Figure 4; the figure also illustrates the banded lower triangular structure of M. Finally, the output matrix is folded back into a tensor. The fold operation is defined to be the inverse of the unfold operation. • The Bitcoin Alpha dataset is available at https://snap.stanford.edu/data/soc-sign-bitcoin-alpha.html. • The Bitcoin OTC dataset is available at https://snap.stanford.edu/data/soc-sign-bitcoin-otc.html. • The Reddit dataset is available at https://snap.stanford.edu/data/soc-RedditHyperlinks.html. Note that we use the dataset with hyperlinks in the body of the posts. • The chess dataset is available at http://konect.uni-koblenz.de/networks/chess. When partitioning the data into T graphs, as described in Section 5, if there are multiple data points corresponding to an edge (m, n) for a given time step t, we only add that edge once to the corresponding graph and set the label equal to the sum of the labels of the different data points. For example, if bitcoin user m makes three transactions to n during time step t with ratings 10, 2, −1, then we add a single edge (m, n) to graph t with label 10 + 2 − 1 = 11. For training, we run gradient descent with a learning rate of 0.01 and momentum of 0.9 for 10,000 iterations. For each 100 iterations, we compute and store the performance of the model on the validation data. As mentioned in Section 5, the weight vector α in the loss function is treated as a hyperparameter in the bitcoin and Reddit experiments. Since these datasets all have two edge classes, let α 0 and α 1 be the weights of the minority (negative) and majority (positive) classes, respectively. Since these parameters add to 1, we have α 1 = 1 − α 0. For all methods, we repeat the bitcoin and Reddit experiments once for each α 0 ∈ {0.75, 0.76, . . ., 0.95}. For each model and dataset, we then find the best stored performance of the model on the validation data across all α 0 values. We then treat the corresponding model as the trained model, and report its performance on the testing data in Tables 3 and 4. The for the chess experiment are computed in the same way, but only for a single vector α = . Throughout this section, · will denote the Frobenius norm (i.e., the square root of the sum of the elements squared) of a matrix or tensor, and · 2 will denote the matrix spectral norm. We first provide a few further that clarify the algebraic properties of the M-product. Let R 1×1×T denote the set of 1 × 1 × T tensors. Similarly, let R N ×1×T denote the set of N × 1 × T tensors. Under the M-product framework, the set R 1×1×T play a role similar to that played by scalars in matrix algebra. With this in mind, the set R N ×1×T can be seen as a length N vector consisting of tubal elements of length T. Propositions D.1 and D.2 make this more precise. Proposition D.1 (Proposition 4.2 in). The set R 1×1×T with product, which is denoted by (, R 1×1×T), is a commutative ring with identity. Proposition D.2 (Theorem 4.1 in). The set R N ×1×T with product, which is denoted by (, R N ×1×T), is a free module over the ring (, R 1×1×T). A free module is similar to a vector space. Like a vector space, it has a basis. Proposition D.3 shows that the lateral slices of Q in the tensor eigendecomposition form a basis for (, R N ×1×T), similarly to how the eigenvectors in a matrix eigendecomposition form a basis. Proposition D.3. The lateral slices Q:n: ∈ R N ×1×T of Q in Definition 3.8 form a basis for (, R N ×1×T). Proof. Let X ∈ R N ×1×T. Note that Since each frontal face of Q × 3 M is an invertible matrix, this implies that each frontal face of S × 3 M is zero, and hence S = 0. So the lateral slices of Q are also linearly independent in (, R N ×1×T)."

Empirical and theoretical study of the effects of staleness in non-synchronous execution on machine learning algorithms.

Three factors (batch size, learning rate, gradient noise) change in predictable way the properties (e.g. sharpness) of minima found by SGD.