Question:
represent the scientific article to find a short tldr
------
E.g. given 'Continuous Bag of Words (CBOW) is a powerful text embedding method. Due to its strong capabilities to encode word content, CBOW embeddings perform well on a wide range of downstream tasks while being efficient to compute. However, CBOW is not capable of capturing the word order. The reason is that the computation of CBOW's word embeddings is commutative, i.e., embeddings of XYZ and ZYX are the same. In order to address this shortcoming, we propose a learning algorithm for the Continuous Matrix Space Model, which we call Continual Multiplication of Words (CMOW). Our algorithm is an adaptation of word2vec, so that it can be trained on large quantities of unlabeled text. We empirically show that CMOW better captures linguistic properties, but it is inferior to CBOW in memorizing word content. Motivated by these findings, we propose a hybrid model that combines the strengths of CBOW and CMOW. Our show that the hybrid CBOW-CMOW-model retains CBOW's strong ability to memorize word content while at the same time substantially improving its ability to encode other linguistic information by 8%. As a , the hybrid also performs better on 8 out of 11 supervised downstream tasks with an average improvement of 1.2%. Word embeddings are perceived as one of the most impactful contributions from unsupervised representation learning to natural language processing from the past few years BID10. Word embeddings are learned once on a large-scale stream of words. A key benefit is that these pre-computed vectors can be re-used almost universally in many different downstream applications. Recently, there has been increasing interest in learning universal sentence embeddings. BID24 have shown that the best encoding architectures are based on recurrent neural networks (RNNs) BID5 BID25 or the Transformer architecture BID2. These techniques are, however, substantially more expensive to train and apply than word embeddings BID14 BID2. Their usefulness is therefore limited when fast processing of large volumes of data is critical. More efficient encoding techniques are typically based on aggregated word embeddings such as Continuous Bag of Words (CBOW), which is a mere summation of the word vectors BID19. Despite CBOW's simplicity, it attains strong on many downstream tasks. Using sophisticated weighting schemes, the performance of aggregated word embeddings can be further increased BID0, coming even close to strong LSTM baselines BID26 BID13 such as InferSent BID5. This raises the question how much benefit recurrent encoders actually provide over simple word embedding based methods BID32. In their analysis, BID13 suggest that the main difference may be the ability to encode word order. In this paper, we propose an intuitive method to enhance aggregated word embeddings by word order awareness. The major drawback of these CBOW-like approaches is that they are solely based on addition. However, addition is not all you need. Since it is a commutative operation, the aforementioned methods are not able to capture any notion of word order. However, word order information is crucial for some tasks, e.g., sentiment analysis BID13. For instance, the following two sentences yield the exact same embedding in an addition-based word embedding aggregation technique: "The movie was not awful, it was rather great." and "The movie was not great, it was rather awful." A classifier based on the CBOW embedding of these sentences would inevitably fail to distinguish the two different meanings .To alleviate this drawback, BID28 propose to model each word as a matrix rather than a vector, and compose multiple word embeddings via matrix multiplication rather than addition. This so-called Compositional Matrix Space Model (CMSM) of language has powerful theoretical properties that subsume properties from vector-based models and symbolic approaches. The most obvious advantage is the non-commutativity of matrix multiplication as opposed to addition, which in order-aware encodings. In contrast to vector-based word embeddings, there is so far no solution to effectively train the parameters of word matrices on large-scale unlabeled data. Training schemes from previous work were specifically designed for sentiment analysis BID34 BID1. Those require complex, multi-stage initialization, which indicates the difficulty of training CMSMs. We show that CMSMs can be trained in a similar way as the well-known CBOW model of word2vec BID19. We make two simple yet critical changes to the initialization strategy and training objective of CBOW. Hence, we present the first unsupervised training scheme for CMSMs, which we call Continual Multiplication Of Words (CMOW). We evaluate our model's capability to capture linguistic properties in the encoded text. We find that CMOW and CBOW have properties that are complementary. On the one hand, CBOW yields much stronger at the word content memorization task. CMOW, on the other hand, offers an advantage in all other linguistic probing tasks, often by a wide margin. Thus, we propose a hybrid model to jointly learn the word vectors of CBOW and the word matrices for CMOW.Our experimental confirm the effectiveness of our hybrid CBOW-CMOW approach. At comparable embedding size, CBOW-CMOW retains CBOW's ability to memorize word content while at the same time improves the performance on the linguistic probing tasks by 8%. CBOW-CMOW outperforms CBOW at 8 out of 11 supervised downstream tasks scoring only 0.6% lower on the tasks where CBOW is slightly better. On average, the hybrid model improves the performance over CBOW by 1.2% on supervised downstream tasks, and by 0.5% on the unsupervised tasks. In summary, our contributions are: For the first time, we present an unsupervised, efficient training scheme for the Compositional Matrix Space Model. Key elements of our scheme are an initialization strategy and training objective that are specifically designed for training CMSMs. We quantitatively demonstrate that the strengths of the ing embedding model are complementary to classical CBOW embeddings. We successfully combine both approaches into a hybrid model that is superior to its individual parts. After giving a brief overview of the related work, we formally introduce CBOW, CMOW, and the hybrid model in Section 3. We describe our experimental setup and present the in Section 4. The are discussed in Section 5, before we conclude. We present an algorithm for learning the weights of the Compositional Matrix Space Model BID28. To the best of our knowledge, only BID34 and BID1 have addressed this. They present complex, multi-level initialization strategies to achieve reasonable . Both papers train and evaluate their model on sentiment analysis datasets only, but they do not evaluate their CMSM as a general-purpose sentence encoder. Other works have represented words as matrices as well, but unlike our work not within the framework of the CMSM. BID11 represent only relational words as matrices. BID29 and BID3 argue that while CMSMs are arguably more expressive than embeddings located in a vector space, the associativeness of matrix multiplication does not reflect the hierarchical structure of language. Instead, they represent the word sequence as a tree structure. BID29 directly represent each word as a matrix (and a vector) in a recursive neural network. BID3 present a two-layer architecture. In the first layer, pre-trained word embeddings are mapped to their matrix representation. In the second layer, a non-linear function composes the constituents. Sentence embeddings have recently become an active field of research. A desirable property of the embeddings is that the encoded knowledge is useful in a variety of high-level downstream tasks. To this end, BID4 and introduced an evaluation framework for sentence encoders that tests both their performance on downstream tasks as well as their ability to capture linguistic properties. Most works focus on either i) the ability of encoders to capture appropriate semantics or on ii) training objectives that give the encoders incentive to capture those semantics. Regarding the former, large RNNs are by far the most popular BID5 BID16 BID31 BID22 BID14 BID18 BID25 BID17, followed by convolutional neural networks BID7. A third group are efficient methods that aggregate word embeddings BID33 BID0 BID23 BID26. Most of the methods in the latter group are word order agnostic. Sent2Vec BID23 is an exception in the sense that they also incorporate bigrams. Despite also employing an objective similar to CBOW, their work is very different to ours in that they still use addition as composition function. Regarding the training objectives, there is an ongoing debate whether language modeling BID25 BID27, machine translation BID18, natural language inference BID5, paraphrase identification BID33, or a mix of many tasks BID30 ) is most appropriate for incentivizing the models to learn important aspects of language. In our study, we focus on adapting the well-known objective from word2vec BID19 for the CMSM. We formally present CBOW and CMOW encoders in a unified framework. Subsequently, we discuss the training objective, the initialization strategy, and the hybrid model. We start with a lookup table for the word matrices, i.e., an embedding, E ∈ R m×d×d, where m is the vocabulary size and d is the dimensionality of the (square) matrices. We denote a specific word matrix of the embedding by E[·]. By ∆ ∈ {,} we denote the function that aggregates word embeddings into a sentence embedding. Formally, given a sequence s of arbitrary length n, the sequence is encoded as DISPLAYFORM0 For ∆ =, the model becomes CBOW. By setting ∆ = (matrix multiplication), we obtain CMOW. Because the of the aggregation for any prefix of the sequence is again a square matrix of shape d × d irrespective of the aggregation function, the model is well defined for any non-zero sequence length. Thus, it can serve as a general-purpose text encoder. Throughout the remainder of this paper, we denote the encoding step by enc DISPLAYFORM1, where flatten concatenates the columns of the matrices to obtain a vector that can be passed to the next layer. Motivated by its success, we employ a similar training objective as word2vec BID20. The objective consists of maximizing the conditional probability of a word w O in a certain context s: p(w O | s). For a word w t at position t within a sentence, we consider the window of tokens (w t−c, . . ., w t+c) around that word. From that window, a target word w O:= {w t+i}, i ∈ {−c, . . ., +c} is selected. The remaining 2c words in the window are used as the context s. The training itself is conducted via negative sampling NEG-k, which is an efficient approximation of the softmax BID20. For each positive example, k negative examples (noise words) are drawn from some noise distribution P n (w). The goal is to distinguish the target word w O from the randomly sampled noise words. Given the encoded input words enc ∆ (s), a logistic regression with weights v ∈ R m×d 2 is conducted to predict 1 for context words and 0 for noise words. The negative sampling training objective becomes: DISPLAYFORM0 In the original word2vec BID19, the center word w O:= w t is used as the target word. In our experiments, however, this objective did not yield to satisfactory . We hypothesize that this objective is too easy to solve for a word order-aware text encoder, which diminishes incentive for the encoder to capture semantic information at the sentence level. Instead, we propose to select a random output word w O ∼ U({w t−c, . . ., w t+c}) from the window. The rationale is the following: By removing the information at which position the word was removed from the window, the model is forced to build a semantically rich representation of the whole sentence. For CMOW, modifying the objective leads to a large improvement on downstream tasks by 20.8% on average, while it does not make a difference for CBOW. We present details in the appendix (Section B.1). So far, only BID34 and BID1 have proposed algorithms for learning the parameters for the matrices in CMSMs. Both works devote particular attention to the initialization, noting that a standard initialization randomly sampled from N (0, 0.1) does not work well due to the optimization problem being non-convex. To alleviate this, the authors of both papers propose rather complicated initialization strategies based on a bag-of-words solution BID34 or incremental training, starting with two word phrases BID1. We instead propose an effective yet simple strategy, in which the embedding matrices are initialized close to the identity matrix. We argue that modern optimizers based on stochastic gradient descent have proven to find good solutions to optimization problems even when those are non-convex as in optimizing the weights of deep neural networks. CMOW is essentially a deep linear neural network with flexible layers, where each layer corresponds to a word in the sentence. The output of the final layer is then used as an embedding for the sentence. A subsequent classifier may expect that all embeddings come from the same distribution. We argue that initializing the weights randomly from N (0, 0.1) or any other distribution that has most of its mass around zero is problematic in such a setting. This includes the Glorot initialization BID8, which was designed to alleviate the problem of vanishing gradients. FIG0 illustrates the problem: With each multiplication, the values in the embedding become smaller (by about one order of magnitude). This leads to the undesirable effect that short sentences have a drastically different representation than larger ones, and that the embedding values vanish for long sequences. To prevent this problem of vanishing values, we propose an initialization strategy, where each word embedding matrix E[w] ∈ R d×d is initialized as a random deviation from the identity matrix: DISPLAYFORM0 It is intuitive and also easy to prove that the expected value of the multiplication of any number of such word embedding matrices is again the identity matrix (see Appendix A). FIG0 shows how our initialization strategy is able to prevent vanishing values. For training CMSMs, we observe a substantial improvement over Glorot initialization of 2.8% on average. We present details in Section B.2 of the appendix. The simplest combination is to train CBOW and CMOW separately and concatenate the ing sentence embeddings at test time. However, we did not find this approach to work well in preliminary experiments. We conjecture that there is still a considerable overlap in the features learned by each model, which hinders better performance on downstream tasks. To prevent redundancy in the learned features, we expose CBOW and CMOW to a shared learning signal by training them jointly. To this end, we modify Equation 1 as follows: DISPLAYFORM1 Intuitively, the model uses logistic regression to predict the missing word from the concatenation of CBOW and CMOW embeddings. Again, E i ∈ R m×di×di are separate word lookup tables for CBOW and CMOW, respectively, and v ∈ R m×(d We conducted experiments to evaluate the effect of using our proposed models for training CMSMs. In this section, we describe the experimental setup and present the on linguistic probing as well as downstream tasks. In order to limit the total batch size and to avoid expensive tokenization steps as much as possible, we created each batch in the following way: 1,024 sentences from the corpus are selected at random. After tokenizing each sentence, we randomly select (without replacement) at maximum 30 words from the sentence to function as center words for a context window of size c = 5, i.e., we generate up to 30 training samples per sentence. By padding with copies of the neutral element, we also include words as center words for which there are not enough words in the left or the right context. For CBOW, the neutral element is the zero matrix. For CMOW, the neutral element is the identity matrix. We trained our models on the unlabeled UMBC news corpus BID12, which consists of about 134 million sentences and 3 billion tokens. Each sentence has 24.8 words on average with a standard deviation of 14.6. Since we only draw 30 samples per sentence to limit the batch size, not all possible training examples are used in an epoch, which may in slightly worse generalization if the model is trained for a fixed number of epochs. We therefore use 0.1% of the 134 million sentences for validation. After 1,000 updates (i.e., approximately every millionth training sample) the validation loss is calculated, and training terminates after 10 consecutive validations of no improvement. Following BID20, we limit the vocabulary to the 30,000 mostfrequent words for comparing our different methods and their variants. Out-of-vocabulary words are discarded. The optimization is carried out by Adam BID15 with an initial learning rate of 0.0003 and k = 20 negative samples as suggested by BID20 for rather small datasets. For the noise distribution P n (w) we again follow BID20 and use U(w) 3/4 /Z, where Z is the partition function to normalize the distribution. We have trained five different models: CBOW and CMOW with d = 20 and d = 28, which lead to 400-dimensional and 784-dimensional word embeddings, respectively. We also trained the Hybrid CBOW-CMOW model with d = 20 for each component, so that the total model has 800 parameters per word in the lookup tables. We report the of two more models: H-CBOW is the 400-dimensional CBOW component trained in Hybrid and H-CMOW is the respective CMOW component. Below, we compare the 800-dimensional Hybrid method to the 784-dimensional CBOW and CMOW models. After training, only the encoder of the model enc E ∆ is retained. We assess the capability to encode linguistic properties by evaluating on 10 linguistic probing tasks. In particular, the Word Content (WC) task tests the ability to memorize exact words in the sentence. Bigram Shift (BShift) analyzes the encoder's sensitivity to word order. The downstream performance is evaluated on 10 supervised and 6 unsupervised tasks from the SentEval framework BID4. We use the standard evaluation configuration, where a logistic regression classifier is trained on top of the embeddings. Considering the linguistic probing tasks (see TAB0), CBOW and CMOW show complementary . While CBOW yields the highest performance at word content memorization, CMOW outperforms CBOW at all other tasks. Most improvements vary between 1-3 percentage points. The difference is approximately 8 points for CoordInv and Length, and even 21 points for BShift. The hybrid model yields scores close to or even above the better model of the two on all tasks. In terms of relative numbers, the hybrid model improves upon CBOW in all probing tasks but WC and SOMO. The relative improvement averaged over all tasks is 8%. Compared to CMOW, the hybrid model shows rather small differences. The largest loss is by 4% on the CoordInv task. However, due to the large gain in WC (20.9%), the overall average gain is still 1.6%.We now compare the jointly trained H-CMOW and H-CBOW with their separately trained 400-dimensional counterparts. We observe that CMOW loses most of its ability to memorize word content, while CBOW shows a slight gain. On the other side, H-CMOW shows, among others, improvements at BShift. TAB1 shows the scores from the supervised downstream tasks. Comparing the 784-dimensional models, again, CBOW and CMOW seem to complement each other. This time, however, CBOW has the upperhand, matching or outperforming CMOW on all supervised downstream tasks except TREC by up to 4 points. On the TREC task, on the other hand, CMOW outperforms CBOW by 2.5 points. Our jointly trained model is not more than 0.8 points below the better one of CBOW and CMOW on any of the considered supervised downstream tasks. On 7 out of 11 supervised tasks, the joint model even improves upon the better model, and on SST2, SST5, and MRPC the difference is more than 1 point. The average relative improvement over all tasks is 1.2%.Regarding the unsupervised downstream tasks TAB2, CBOW is clearly superior to CMOW on all datasets by wide margins. For example, on STS13, CBOW's score is 50% higher. The hybrid model is able to repair this deficit, reducing the difference to 8%. It even outperforms CBOW on two of the tasks, and yields a slight improvement of 0.5% on average over all unsupervised downstream tasks. However, the variance in relative performance is notably larger than on the supervised downstream tasks. Our CMOW model produces sentence embeddings that are approximately at the level of fastSent BID14. Thus, CMOW is a reasonable choice as a sentence encoder. Essential to the success of our training schema for the CMOW model are two changes to the original word2vec training. First, our initialization strategy improved the downstream performance by 2.8% compared to Glorot initialization. Secondly, by choosing the target word of the objective at random, the performance of CMOW on downstream tasks improved by 20.8% on average. Hence, our novel training scheme is the first that provides an effective way to obtain parameters for the Compositional Matrix Space Model of language from unlabeled, large-scale datasets. Regarding the probing tasks, we observe that CMOW embeddings better encode the linguistic properties of sentences than CBOW. CMOW gets reasonably close to CBOW on some downstream tasks. However, CMOW does not in general supersede CBOW embeddings. This can be explained by the fact that CBOW is stronger at word content memorization, which is known to highly correlate with the performance on most downstream tasks ). Yet, CMOW has an increased performance on the TREC question type classification task (88.0 compared to 85.6). The rationale is that this particular TREC task belongs to a class of downstream tasks that require capturing other linguistic properties apart from Word Content.Due to joint training, our hybrid model learns to pick up the best features from CBOW and CMOW simultaneously. It enables both models to focus on their respective strengths. This can best be seen by observing that H-CMOW almost completely loses its ability to memorize word content. In return, H-CMOW has more capacity to learn other properties, as seen in the increase in performance at BShift and others. A complementary behavior can be observed for H-CBOW, whose scores on Word Content are increased. Consequently, with an 8% improvement on average, the hybrid model is substantially more linguistically informed than CBOW. This transfers to an overall performance improvement by 1.2% on average over 11 supervised downstream tasks, with large improvements on sentiment analysis tasks (SST2, SST5), question classification (TREC), and the sentence representation benchmark (STS-B). The improvements on these tasks is expected because they arguably depend on word order information. On the other tasks, the differences are small. Again, this can be explained by the fact that most tasks in the SentEval framework mainly depend on word content memorization, where the hybrid model does not improve upon CBOW.Please note, the models in our study do not represent the state-of-the-art for sentence embeddings. BID24 show that better scores are achieved by LSTMs and Transformer models, but also by averaging word embedding from fastText BID21. These embeddings were trained on the CBOW objective, and are thus very similar to our models. However, they are trained on large corpora (600B tokens vs 3B in our study), use large vocabularies (2M vs 30k in our study), and incorporate numerous tricks to further enhance the quality of their models: word subsampling, subword-information, phrase representation, n-gram representations, position-dependent weighting, and corpus de-duplication. In the present study, we focus on comparing CBOW, CMOW, and the hybrid model in a scenario where we have full control over the independent variables. To single out the effect of the independent variables better, we keep our models relatively simple. Our analysis yields interesting insights on what our models learn when trained separately or jointly, which we consider more valuable in the long term for the research field of text representation learning. We offer an efficient order-aware extension to embedding algorithms from the bag-of-words family. Our 784-dimensional CMOW embeddings can be computed at the same rate as CBOW embeddings. We empirically measured in our experiments 71k for CMOW vs. 61k for CBOW in terms of encoding sentences per second. This is because of the fast implementation of matrix multiplication in GPUs. It allows us to encode sentences approximately 5 times faster than using a simple Elman RNN of the same size (12k per second). Our matrix embedding approach also offers valuable theoretical advantages over RNNs and other autoregressive models. Matrix multiplication is associative such that only log 2 n sequential steps are necessary to encode a sequence of size n. Besides parallelization, also dynamic programming techniques can be employed to further reduce the number of matrix multiplication steps, e. g., by pre-computing frequent bigrams. We therefore expect our matrix embedding approach to be specifically well-suited for large-scale, time-sensitive text encoding applications. Our hybrid model serves as a blueprint for using CMOW in conjunction with other existing embedding techniques such as fastText BID21. We have presented the first efficient, unsupervised learning scheme for the word order aware Compositional Matrix Space Model. We showed that the ing sentence embeddings capture linguistic features that are complementary to CBOW embeddings. We thereupon presented a hybrid model with CBOW that is able to combine the complementary strengths of both models to yield an improved downstream task performance, in particular on tasks that depend on word order information. Thus, our model narrows the gap in terms of representational power between simple word embedding based sentence encoders and highly non-linear recurrent sentence encoders. We made the code for this paper available at https://github.com/florianmai/ word2mat. The statement that we formally proof is the following. For any sequence s = s 1... s n: DISPLAYFORM0 The basis (n = 1) follows trivially due to the expected value of each entry being the mean of the normal distribution. For the induction step, let E[DISPLAYFORM1 In Section 3.2, we describe a more general training objective than the classical CBOW objective from BID19 . The original objective always sets the center word from the window of tokens (w t−c, . . ., w t+c) as target word, w O = w t. In preliminary experiments, this did not yield satisfactory . We believe that this objective is too simple for learning sentence embeddings that capture semantic information. Therefore, we experimented a variant where the target word is sampled randomly from a uniform distribution, w O:= U({w t−c, . . ., w t+c}).To test the effectiveness of this modified objective, we evaluate it with the same experimental setup as described in Section 4. TAB3 lists the on the linguistic probing tasks. CMOW-C and CBOW-C refer to the models where the center word is used as the target. CMOW-R and CBOW-R refer to the models where the target word is sampled randomly. While CMOW-R and CMOW-C perform comparably on most probing tasks, CMOW-C yields 5 points lower scores on WordContent and BigramShift. Consequently, CMOW-R also outperforms CMOW-C on 10 out of 11 supervised downstream tasks and on all unsupervised downstream tasks, as shown in TAB4, respectively. On average over all downstream tasks, the relative improvement is 20.8%. For CBOW, the scores on downstream tasks increase on some tasks and decrease on others. The differences are miniscule. On average over all 16 downstream tasks, CBOW-R scores 0.1% lower than CBOW-C. In Section 3.3, we present a novel random initialization strategy. We argue why it is more adequate for training CMSMs than classic strategies that initialize all parameters with random values close to zero, and use it in our experiments to train CMOW.To verify the effectiveness of our initialization strategy empirically, we evaluate it with the same experimental setup as described in Section 4. The only difference is the initialization strategy, where we include Glorot initialization BID8 and the standard initialization from N (0, 0.1). TAB6 shows the on the probing tasks. While Glorot achieves slightly better on BShift and TopConst, CMOW's ability to memorize word content is improved by a wide margin by our initialization strategy. This again affects the downstream performance as shown in TAB7 and 9, respectively: 7 out of 11 supervised downstream tasks and 4 out of 5 unsupervised downstream tasks improve. On average, the relative improvement of our strategy compared to Glorot initialization is 2.8%.' it should be close to 'We present a novel training scheme for efficiently obtaining order-aware sentence representations.' but not to 'This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network. '.

Question: Experimental reproducibility and replicability are critical topics in machine learning. Authors have often raised concerns about their lack in scientific publications to improve the quality of the field. Recently, the graph representation learning field has attracted the attention of a wide research community, which ed in a large stream of works. As such, several Graph Neural Network models have been developed to effectively tackle graph classification. However, experimental procedures often lack rigorousness and are hardly reproducible. Motivated by this, we provide an overview of common practices that should be avoided to fairly compare with the state of the art. To counter this troubling trend, we ran more than 47000 experiments in a controlled and uniform framework to re-evaluate five popular models across nine common benchmarks. Moreover, by comparing GNNs with structure-agnostic baselines we provide convincing evidence that, on some datasets, structural information has not been exploited yet. We believe that this work can contribute to the development of the graph learning field, by providing a much needed grounding for rigorous evaluations of graph classification models. Over the years, researchers have raised concerns about several flaws in scholarship, such as experimental reproducibility and replicability in machine learning and science in general (National Academies of). These issues are not easy to address, as a collective effort is required to avoid bad practices. Examples include the ambiguity of experimental procedures, the impossibility of reproducing and the improper comparison of machine learning models. As a , it can be difficult to uniformly assess the effectiveness of one method against another. This work investigates these issues for the graph representation learning field, by providing a uniform and rigorous benchmarking of state-of-the-art models. Graph Neural Networks (GNNs) have recently become the standard tool for machine learning on graphs. These architectures effectively combine node features and graph topology to build distributed node representations. GNNs can be used to solve node classification and link prediction tasks, or they can be applied to downstream graph classification . In literature, such models are usually evaluated on chemical and social domains . Given their appeal, an ever increasing number of GNNs is being developed . However, despite the theoretical advancements reached by the latest contributions in the field, we find that the experimental settings are in many cases ambiguous or not reproducible. Some of the most common reproducibility problems we encounter in this field concern hyperparameters selection and the correct usage of data splits for model selection versus model assessment. Moreover, the evaluation code is sometimes missing or incomplete, and experiments are not standardized across different works in terms of node and edge features. These issues easily generate doubts and confusion among practitioners that need a fully transparent and reproducible experimental setting. As a matter of fact, the evaluation of a model goes through two different phases, namely model selection on the validation set and model assessment on the test set. Clearly, to fail in keeping these phases well separated could lead to over-optimistic and biased estimates of the true performance of a model, making it hard for other researchers to present competitive without following the same ambiguous evaluation procedures. With this premise, our primary contribution is to provide the graph learning community with a fair performance comparison among GNN architectures, using a standardized and reproducible experimental environment. More in detail, we performed a large number of experiments within a rigorous model selection and assessment framework, in which all models were compared using the same features and the same data splits. Secondly, we investigate if and to what extent current GNN models can effectively exploit graph structure. To this end, we add two domain-specific and structure-agnostic baselines, whose purpose is to disentangle the contribution of structural information from node features. Much to our surprise, we found out that these baselines can even perform better than GNNs on some datasets; this calls for moderation when reporting improvements that do not clearly outperform structure-agnostic competitors. Our last contribution is a study on the effect of node degrees as features in social datasets. Indeed, we show that providing the degree can be beneficial in terms of performances, and it has also implications in the number of GNN layers needed to reach good . We publicly release code and dataset splits to reproduce our , in order to allow other researchers to carry out rigorous evaluations with minimum additional effort 1. Disclaimer Before delving into the work, we would like to clarify that this work does not aim at pinpointing the best (or worst) performing GNN, nor it disavows the effort researchers have put in the development of these models. Rather, it is intended to be an attempt to set up a standardized and uniform evaluation framework for GNNs, such that future contributions can be compared fairly and objectively with existing architectures. Graph Neural Networks At the core of GNNs is the idea to compute a state for each node in a graph, which is iteratively updated according to the state of neighboring nodes. Thanks to layering or recursive schemes, these models propagate information and construct node representations that can be "aware" of the broader graph structure. GNNs have recently gained popularity because they can efficiently and automatically extract relevant features from a graph; in the past, the most popular way to deal with complex structures was to use kernel functions to compute task-agnostic features. However, such kernels are non-adaptive and typically computationally expensive, which makes GNNs even more appealing. Even though in this work we specifically focus on architectures designed for graph classification, all GNNs share the notion of "convolution" over node neighborhoods, as a generalization of convolution on grids. For example, GraphSAGE first performs sum, mean or max-pooling neighborhood aggregation, and then it updates the node representation applying a linear projection on top of the convolution. It also relies on a neighborhood sampling scheme to keep computational complexity constant. Instead, Graph Isomorphism Network (GIN) builds upon the limitations of GraphSAGE, extending it with arbitrary aggregation functions on multi-sets. The model is proven to be as theoretically powerful as the Weisfeiler-Lehman test of graph isomorphism. Very recently, gave an upper bound to the number of hidden units needed to learn permutation-invariant functions over sets and multi-sets. Differently from the above methods, Edge-Conditioned Convolution (ECC) learns a different parameter for each edge label. Therefore, neighbor aggregation is weighted according to specific edge parameters. Finally, Deep Graph Convolutional Neural Network (DGCNN) proposes a convolutional layer similar to the formulation of. Some models also exploit a pooling scheme, which is applied after convolutional layers in order to reduce the size of a graph. For example, the pooling scheme of ECC coarsens graphs through a differentiable pooling map that can be pre-computed. Similarly, DiffPool proposes an adaptive pooling mechanism that collapses nodes on the basis of a supervised criterion. In practice, DiffPool combines a differentiable graph encoder with its pooling strategy, so that the architecture is end-to-end trainable. Lastly, DGCNN differs from other works in that nodes are sorted and aligned by a specific algorithm called SortPool. Model evaluation The work of shares a similar purpose with our contribution. In particular, the authors compare different GNNs on node classification tasks, showing that are highly dependent on the particular train/validation/test split of choice, up to the point where changing splits leads to dramatically different performance rankings. Thus, they recommend to evaluate GNNs on multiple test splits to achieve a fair comparison. Even though we operate in a different setting (graph instead of node classification), we follow the authors' suggestions by evaluating models under a controlled and rigorous assessment framework. Finally, the work of criticizes a large number of neural recommender systems, most of which are not reproducible, showing that only one of them truly improves against a simple baseline. Here, we recap the risk assessment (also called model evaluation or model assessment) and model selection procedures, to clearly layout the experimental procedure followed in this paper. For space reasons, the overall procedure is visually summarized in Appendix A.1. The goal of risk assessment is to provide an estimate of the performance of a class of models. When a test set is not explicitly given, a common way to proceed is to use k-fold Cross Validation (CV) (; ;). k-fold CV uses k different training/test splits to estimate the generalization performance of a model; for each partition, an internal model selection procedure selects the hyper-parameters using the training data only. This way, test data is never used for model selection. As model selection is performed independently for each training/test split, we obtain different "best" hyper-parameter configurations; this is why we refer to the performance of a class of models. The goal of model selection, or hyper-parameter tuning, is to choose among a set of candidate hyperparameter configurations the one that works best on a specific validation set. If a validation set is not given, one can rely on a holdout training/validation split or an inner k-fold. Nevertheless, the key point to remember is that validation performances are biased estimates of the true generalization capabilities. Consequently, model selection are generally over-optimistic; this issue is thoroughly documented in. This is why the main contribution of this work is to clearly separate model selection and model assessment estimates, something that is lacking or ambiguous in the literature under consideration. To motivate our contribution, we follow the approach of and briefly review recent papers describing five different GNN models, highlighting problems in the experimental setups as well as reproducibility of . We emphasize that our observations are based solely on the contents of their paper and the available code 2. Suitable GNN works were selected according to the following criteria: i) performances obtained with 10-fold CV; ii) peer reviewed; iii) strong architectural differences; iv) popularity. In particular, we selected DGCNN, DiffPool , ECC , GIN and GraphSAGE . For a detailed description of each model we refer to their respective papers. Our criteria to assess quality of evaluation and reproducibility are: i) code for data preprocessing, model selection and assessment is provided; ii) data splits are provided; iii) data is split by means of a stratification technique, to preserve class proportions across all partitions; iv) of the 10-fold CV are reported correctly using standard deviations, and they refer to model evaluation (test sets) rather than model selection (validation sets). Table 1 summarizes our findings. Table 1: Criteria for reproducibility considered in this work and their compliance among considered models. (Y) indicates that the criterion is met, (N) indicates that the criterion is not satisfied, (A) indicates ambiguity (i.e. it is unclear whether the criteria is met or not), (-) indicates lack of information (i.e. no details are provided about the criteria). Note that GraphSAGE is excluded from this comparison, as it was not directly applied by authors to graph classification tasks. The authors evaluate the model on 10-fold CV. While the architecture is fixed for all dataset, learning rate and epochs are tuned using only one random CV fold, and then reused on all the other folds. While this practice is still acceptable, it may lead to sub-optimal performances. Nonetheless, the code to reproduce model selection is not available. Moreover, the authors run CV 10 times, and they report the average of the 10 final scores. As a , the variance of the provided estimates is reduced. However, the same procedure was not applied to the other competitors as well. Finally, CV data splits are correctly stratified and publicly available, making it possible to reproduce at least the evaluation experiments. DiffPool From both the paper and the provided code, it is unclear if reported are obtained on a test set rather than a validation set. Although the authors state that 10-fold CV is used, standard deviations of DiffPool and its competitors are not reported. Moreover, the authors affirm to have applied early stopping on the validation set to prevent overfitting; unfortunately, neither model selection code nor validation splits are available. Furthermore, according to the code, data is randomly split (without stratification) and no random seed is set, hence splits are different each time the code is executed. ECC The paper reports that ECC is evaluated on 10-fold CV, but do not include standard deviations. Similarly to DGCNN, hyper-parameters are fixed in advance, hence it is not clear if and how model selection has been performed. Importantly, there are no references in the code repository to data pre-processing, data stratification, data splitting, and model selection. GIN The authors correctly list all the hyper-parameters tuned. However, as stated explicitly in the paper and in the public review discussion, they report the validation accuracy of 10-fold CV. In other words, reported refer to model selection and not to model evaluation. The code for model selection is not provided. GraphSAGE The original paper does not test this model on graph classification datasets, but GraphSAGE is often used in other papers as a strong baseline. It follows that GraphSAGE on graph classification should be accompanied by the code to reproduce the experiments. Despite that, the two works which report of GraphSAGE (DiffPool and GIN) fail to do so. Summary Our analysis reveals that GNN works rarely comply with good machine learning practices as regards the quality of evaluation and reproducibility of . This motivates the need to re-evaluate all models within a rigorous, reproducible and fair environment. In this section we detail our main experiment, in which we re-evaluate the above-mentioned models on 9 datasets (4 chemical, 5 social), using a model selection and assessment framework that closely follows the rigorous practices described in Section 3. In addition, we implement two baselines whose purpose is to understand the extent to which GNNs are able to exploit structural information. All models have been implemented by means of the Pytorch Geometrics library , which provides graph pre-processing routines and makes the definition of graph convolution easier to implement. We sometimes found discrepancies between papers and related code; in such cases, we complied with the specifications in the paper. Because GraphSAGE was not applied to graph classification in the original work, we opted for a max-pooling global aggregation function to classify graph instances; further, we do not use the sampled neighborhood aggregation scheme defined in , in order to allow nodes to have access to their whole neighborhood. Datasets All graph datasets are publicly available and represent a relevant subset of those most frequently used in literature to compare GNNs. Some collect molecular graphs, while others contain social graphs. In particular, we used D&D , PROTEINS , NCI1 and ENZYMES for binary and multi-class classification of chemical compounds, whereas IMDB-BINARY, IMDB-MULTI, REDDIT-BINARY, REDDIT-5K and COLLAB are social datasets. Dataset statistics are reported in Table A.2. Features In GNN literature, it is common practice to augment node descriptors with structural features. For example, DiffPool adds the degree and clustering coefficient to each node feature vector, whereas GIN adds a one-hot representation of node degrees. The latter choice trades off an improvement in performances (due to injectivity of the first sum) with the inability to generalize to graphs with arbitrary node degree. In general, good experimental practices suggest that all models should be consistently compared to the same input representations. This is why we re-evaluate all models using the same node features. In particular, we use one common setting for the chemical domain and two alternative settings as regards the social domain. As regards the chemical domain, nodes are labeled with a one-hot encoding of their atom type, though on ENZYMES we follow the literature and use 18 additional features available. As regards social graphs, whose nodes do not have features, we use either an uninformative feature for all nodes or the node degree. As such, we are able to reason about the effectiveness of the structural inductive bias imposed by the model; that is if the model is able to implicitly learn structural features or not. The effect of adding structural features to general machine learning models for graphs has been investigated in; here, we focus on the impact of node degrees on performances for social datasets. Baselines We adopt two distinct baselines, one for chemical and one for social datasets. On all chemical datasets but for ENZYMES, we follow; and implement the Molecular Fingerprint technique, which first applies global sum pooling (i.e., counts the occurrences of atom types in the graph by summing the features of all nodes in the graph together) and then applies a single-layer MLP with ReLU activations. On social domains and ENZYMES (due to the presence of additional features), we take inspiration from the work of to learn permutation-invariant functions over sets of nodes: first, we apply a single-layer MLP on top of node features, followed by global sum pooling and another singlelayer MLP for classification. Note that both baselines do not leverage graph topology. Using these baselines as a reference is of fundamental importance for future works, as they can provide feedback on the effectiveness of GNNs on a specific dataset. As a matter of fact, if GNN performances are close to the ones of a structure-agnostic baseline, one can draw two possible : the task does not need topological information to be effectively solved, or the GNN is not exploiting graph structure adequately. While the former can be verified through domain-specific human expertise, the second is more difficult to assess, as multiple factors come into play such as the amount of training data, the structural inductive bias imposed by the architecture and the hyper-parameters used for model selection. Nevertheless, significant improvements with respect to these baselines are a strong indicator that graph topology has been exploited. Therefore, structure-agnostic baselines become vital to understand if and how a model can be improved. Table 2: Pseudo-code for model assessment (left) and model selection (right). In Algorithm 1, "Select" refers to Algorithm 2, whereas "Train" and "Eval" represent training and inference phases, respectively. After each model selection, the best configuration best k is used to evaluate the external test fold. Performances are averaged across R training runs, where R in our case is set to 3. for r ← 1,..., R do Experimental Setting Our experimental approach is to use a 10-fold CV for model assessment and an inner holdout technique with a 90%/10% training/validation split for model selection. After each model selection, we train three times on the whole training fold, holding out a random fraction (10%) of the data to perform early stopping. These three separate runs are needed to smooth the effect of unfavorable random weight initialization on test performances. The final test fold score is obtained as the mean of these three runs; Table 2 reports the pseudo-code of the entire evaluation process. To be consistent with literature, we implement early stopping with patience parameter n, where training stops if n epochs have passed without improvement on the validation set. A high value of n can favor model selection by making it less sensitive to fluctuations in the validation score at the cost of additional computation. Importantly, all data partitions have been pre-computed, so that models are selected and evaluated on the same data splits. Moreover, all data splits are stratified, i.e., class proportions are preserved inside each k-fold split as well as in the holdout splits used for model selection. Hyper-parameters Hyper-parameter tuning is performed via grid search. For the sake of conciseness, we list all hyper-parameters in Section A.4. Notice that we always include those used by other authors in their respective papers. We select the number of convolutional layers, the embedding space dimension, the learning rate, and the criterion for early stopping (either based on the validation accuracy or validation loss) for all models. Depending on the model, we also selected regularization terms, dropout, and other model-specific parameters. Computational considerations Our experiments involve a large number of training runs. For all models, grid sizes range from 32 to 72 possible configurations, depending on the number of hyper-parameters to choose from. However, we tried to keep the upper bound on the number of parameters as similar as possible across models. The total effort required, in terms of the number of single training runs, to complete model assessment procedures exceeded 47000. Such a large number required extensive use of parallelism, both in CPU and GPU, to conduct the experiments in a reasonable amount of time. We emphasize that in some cases (e.g. ECC in social datasets), training on a single hyper-parameter configuration required more than 72 hours, which would have made the sequential exploration of one single grid last months. Therefore, due to the large amount of experiments to conduct and to the computational resources available, we limited the time to complete a single training to 72 hours. 65.2 ± 6.4 DGCNN 76.6 ± 4.3 76.4 ± 1.7 72.9 ± 3.5 38.9 ± 5.7 DiffPool 75.0 ± 3.5 76.9 ± 1.9 73.7 ± 3.5 59.5 ± 5.6 ECC 72.6 ± 4.1 76.2 ± 1.4 72.3 ± 3.4 29.5 ± 8.2 GIN 75.3 ± 2.9 80.0 ± 1.4 73.3 ± 4.0 59.6 ± 4.5 GraphSAGE 72.9 ± 2.0 76.0 ± 1.8 73.0 ± 4.5 58.2 ± 6.0 Tables 3 and 4 show the of our experiments. Overall, GIN seems to be effective on social datasets. Importantly, we discover that on D&D, PROTEINS and ENZYMES none of the GNNs are able to improve over the baseline. On the contrary, on NCI1 the baseline is clearly outperformed: this suggests that the GNNs we analyzed can actually exploit the topological information of the graphs in this dataset. Moreover, we observe that an overly-parameterized baseline is not able to overfit the NCI1 training data completely. To see this, consider that a baseline with 10000 hidden units and no regularization reaches around 67% training accuracy, while GIN can easily overfit (≈ 100%) the training data. This indicates that structural information hugely affects the ability to fit the training set. On social datasets, we observe that adding node degrees as features is beneficial, but such an effect is more noticeable for REDDIT-BINARY, REDDIT-5K and COLLAB. Our also show that structure-agnostic baselines are an essential tool to understand the effectiveness of GNNs and extract useful insights. As an example, since none of the GNNs surpasses the baseline on D&D, PROTEINS and ENZYMES, we argue that the state-of-the-art GNN models we analyzed are not able to fully exploit the structure on such datasets yet; indeed, in chemistry, structural features are known to correlate with molecular properties (van). For all these reasons, we suggest putting small performance gains on these datasets into the right perspective, at least until the baseline will clearly be outperformed. Currently, small average fluctuations on these datasets are likely to be caused by other factors, such as random initializations, rather than a successful exploitation of the structure. In , we warmly recommend GNN practitioners to include baseline comparisons in future works, in order to better characterize the extent of their contributions. Based on our , using node degrees as input features is almost always beneficial to increase performances on social datasets, sometimes by a large amount. As an example, degree information is sufficient for our baseline to improve performances of ≈ 15%, hence being competitive on many datasets; in particular, the baseline achieves the best performance on IMDB-BINARY. In contrast, adding node degrees is less relevant for most GNNs, since they can automatically infer such information from the structure. One notable exception is DGCNN, which explicitly needs node degrees to perform well on all datasets. Moreover, we observe that the ranking of all models, after the addition of the degrees, drastically changes; this raises the question about the impact of other structural features (such as clustering coefficient) on performances, which we leave to future works. However, one may also wonder whether the addition of the degree has an influence on the number of layers that are necessary to solve the task or not. We therefore investigated the matter by computing the median number of layers across the 10 different folds. We observed a general trend across models, with GraphSAGE being the only exception, where the addition of the degree reduces the number of layers needed by ≈ 1 as shown in Table A. 3. This may be due to the fact that most architectures find useful to compute the degree at the very first layer, as such information seems useful to the overall performances. 71.2 ± 3.9 48.5 ± 3.3 89.9 ± 1.9 56.1 ± 1.7 75.6 ± 2.3 GraphSAGE 68.8 ± 4.5 47.6 ± 3.5 84.3 ± 1.9 50.0 ± 1. between the two estimates is usually consistent. In contrast, our average validation accuracies are always higher or equal to our test ; this is expected, as discussed in Section 3.2. Finally, we emphasize once again that our are i) obtained within the framework of a rigorous model selection and assessment protocol; ii) fair with respect of data splits and input features assigned to all competitors; iii) reproducible. In contrast, we saw in Section 4 how published rely on unclear or poorly documented experimental settings. In this paper, we wanted to show how a rigorous empirical evaluation of GNNs can help design future experiments and better reason about the effectiveness of different architectural choices. To this aim, we highlighted ambiguities in the experimental settings of different papers, and we proposed a clear and reproducible procedure for future comparisons. We then provided a complete re-evaluation of five GNNs on nine datasets, which required a significant amount of time and computational resources. This uniform environment helped us reason about the role of structure, as we found that structure-agnostic baselines outperform GNNs on some chemical datasets, thus suggesting that structural properties have not been exploited yet. Moreover, we objectively analyzed the effect of the degree feature on performances and model selection in social datasets, unveiling an effect on the depth of GNNs. Finally, we provide the graph learning community with reliable and reproducible to which GNN practitioners can compare their architectures. We hope that this work, along with the library we release, will prove useful to researchers and practitioners that want to compare GNNs in a more rigorous way. A.1 VISUALIZATION OF THE EVALUATION FRAMEWORK Figure 2: We give a visual representation of the evaluation framework. We apply an external k outfold CV to get an estimate of the generalization performance of a model, and we use an hold-out technique (bottom-left) to select the best hyper-parametres. For completeness, we show that it is also possible to apply an inner k inn -fold CV (implementing a complete Nested Cross Validation), which obviously amounts to multiplying the computational costs of model selection by a factor k inn. A.4 HYPER-PARAMETERS TABLE Table 7: Hyper-parameters used for model selection.

We provide a rigorous comparison of different Graph Neural Networks for graph classification.

Represent the TLDR for finding the corresponding scientific article

We present a new approach, SNIP, that is simple, versatile and interpretable; it prunes irrelevant connections for a given task at single-shot prior to training and is applicable to a variety of neural network models without modifications.


Question: Represent the text to find a very short summary (1-2 sentences max)!

Question: Data augmentation (DA) is fundamental against overfitting in large convolutional neural networks, especially with a limited training dataset. In images, DA is usually based on heuristic transformations, like geometric or color transformations. Instead of using predefined transformations, our work learns data augmentation directly from the training data by learning to transform images with an encoder-decoder architecture combined with a spatial transformer network. The transformed images still belong to the same class, but are new, more complex samples for the classifier. Our experiments show that our approach is better than previous generative data augmentation methods, and comparable to predefined transformation methods when training an image classifier. Convolutional neural networks have shown impressive in visual recognition tasks. However, for proper training and good performance, they require large labeled datasets. If the amount of training data is small, data augmentation is an effective way to improve the final performance of the network BID6; BID9 ). In images, data augmentation (DA) consists of applying predefined transformations such as flip, rotations or color changes BID8; BID3 ). This approach provides consistent improvements when training a classifier. However, the required transformations are dataset dependent. For instance, flipping an image horizontally makes sense for natural images, but produces ambiguities on datasets of numbers (e.g. 2 and 5).Several recent studies investigate automatic DA learning as a method to avoid the manual selection of transformations. BID10 define a large set of transformations and learn how to combine them. This approach works well however, as it is based on predefined transformations, it prevents the model from finding other transformations that could be useful for the classifier. Alternatively, BID2 and BID12 generate new samples via a generative adversarial networks model (GAN) from the probability distribution of the data p(X), while BID0 learn the transformations of images, instead of generating images from scratch. These alternative methods show their limits when the number of training samples is low, given the difficulty of training a high-performing generative model with a reduced dataset. BID5 learn the natural transformations in a dataset by aligning pairs of samples from the same class. This approach produces good on easy datasets like MNIST however, it does not appear to be applicable to more complex datasets. Our work combines the advantages of generative models and transformation learning approaches in a single end-to-end network architecture. Our model is based on a conditional GAN architecture that learns to generate transformations of a given image that are useful for DA. In other words, instead of learning to generate samples from p(X), it learns to generate samples from the conditional distribution p(X|X), withX a reference image. As shown in FIG0, our approach combines a global transformation defined by an affine matrix with a more localized transformation defined by ensures that the transformed sample G(x i, z) is dissimilar from the input sample x i but similar to a sample x j from the same class. (b) Given an input image x i and a random noise vector z, our generator first performs a global transformation using a spatial transformer network followed by more localized transformations using a convolutional encoder-decoder network.a convolutional encoder-decoder architecture. The global transformations are learned by an adaptation of spatial transformer network (STN) BID7 ) so that the entire architecture is differentiable and can be learned with standard back-propagation. In its normal use, the purpose of STN is to learn how to transform the input data, so that the model becomes invariant to certain transformations. In contrast, our approach uses STN to generate augmented samples in an adversarial way. With the proposed model we show that, for optimal performance, it is important to jointly train the generator of the augmented samples with the classifier in an end-to-end fashion. By doing that, we can also add an adversarial loss between the generator and classifier such that the generated samples are difficult, or adversarial, for the classifier. To summarize, the contributions of this paper are: i) We propose a DA network that can automatically learn to generate augmented samples without expensive searches for the optimal data transformations; ii) Our model trains jointly with a classifier, is fully differentiable, trainable end-to-end, and can significantly improve the performance of any image classifier; iii) In low-data regime it outperforms models trained with strong predefined DA; iv) Finally, we notice that, for optimal performance, it is fundamental to train the model jointly with the image classifier. We propose a GAN based architecture that learns to augment training data for image classification. As shown in FIG0, this architecture involves four modules: a generator to transform an input image, two discriminators and a classifier to perform the final classification task. In FIG0 we show the structure of the generator. Instead of generating a new image, as in most GAN models, our generator learns to transform the input image. Our intuition is that learning an image transformation instead of learning a mapping from noise to image is an easier task in low data regime. Given an input image x i and a noise vector z, E N C converts them into a small representation that is passed to T to generate an image and noise dependent affine transformation (similar to spatial transformer networks STN) of the original image. This transformed image is then passed to a U-Net network BID11 ) represented by E N C and D EC. While in the original paper STN was used for removing invariances from the input data, the proposed model generates samples with transformations that can help to learn a better classifier. The generator is supported by two discriminators. The first one, the class discriminator D C, ensures that the generated sample belongs to the same class as the input sample. The second one, the dissimilarity discriminator D D, forces the transformed sample to be dissimilar to the input sample but similar to a sample from the same class. This is necessary to prevent the generator from learning the identity transformation. More details about the different networks and the loss functions used to train the model can be found in Appendices A and B. In this section, we present several experiments to better understand our model and compare it with the state-of-the-art in automatic DA. We test our approach on MNIST, Fashion-MNIST, SVHN, and CIFAR-10, both in full dataset and low-data regime. In this series of experiment, we compare the efficiency of the DA learned by our model to a heuristically chosen DA. We consider two different levels of DA. Light DA refers to random padding of 4 pixels on each side of the image, followed by a crop back to the original image dimensions. Strong DA adds to the previous transformations also rotation in range [-10, 10] degrees and scaling, with factor in range [0.5, 2]. For CIFAR10, DA also includes a horizontal image flip. In a first experiment, we compare the accuracy of the baseline classifier, the baseline with DA, and our DA model while increasing the number of training samples. In our model, the classifier is trained jointly with the generator. In FIG1, we notice that for very few samples the predefined DA is still better than our approach. This is probably because when the training dataset is too small, the generator produces poor samples that are not helpful for the classifier. When the number of samples increases, our approach obtains a much better accuracy than strong DA. For instance, at 4000 training samples, the baseline obtains an accuracy of 66%, the predefined DA approach 76%, and our model 80.5%, thus a net gain of 14 points compared to the baseline and 4 points compared to strong DA model. If we add more examples, the gap between our learned DA and the strong DA tends to reduce. With the full dataset we reach about a half point better than the strong DA.In a second experiment, we compare different types of DA on four datasets with a reduced number of samples. As shown in Tab. 1, our best model is always performing better than light DA and strong DA. This means that our DA model learns transformations that are more useful for the final classifier. Notice that in FMNIST light DA decreases performance of the final classifier. This suggests that DA is dataset dependent and transformations producing useful new samples in some domains might not be usable in others. In this experiment, we report the performance of our method on Joint training and Separate training and compare them with a Baseline model trained without DA. In Joint training the generator of augmented images and the classifier are trained simultaneously in an end-to-end training. In Separate training instead, the generator is first trained to generate augmented images, and these images are then used as DA to improve the classifier. In FIG2, we notice the different behavior of the two methods. In the early phase of training, at epoch 200, both Separate training (beige bar) and the Joint training (red bar) perform above 70%, whereas Baseline has a much lower accuracy. However, with additional training epochs, the performance of Separate training decreases while Baseline and Joint training accuracies increase. We believe that for good performance in DA it is not just about generating plausible augmented samples, but also about generating the right samples at the right moment, as in curriculum learning BID1 FORMULA0 ), our method obtains slightly lower accuracies. However, TANDA is based on the selection of multiple predefined transformations. This means that its learning is reduced to a set of manually selected transformation, which, we believe, reduces the search space and facilitates the task. Also, TANDA uses an additional standard DA based on image crop, while our method does not need any additional DA. On the other hand, our method compares favorably to Bayesian DA BID12 ) and DADA BID13 ), both based on GAN models with a larger neural network for the classifier. This shows that our combination of global and local transformations helps to improve the final performance of the method. In this work, we have presented a new approach for improving the learning of a classifier through an automatic generation of augmented samples. The method is fully differentiable and can be learned end-to-end. In our experiments, we have shown several elements contributing to an improved classification performance. First, the generator and the classifier should be trained jointly. Second, the combined use of global transformations with STN and local transformation with U-Net is essential to reach the highest accuracy levels. For future work, we want to include more differentiable transformations such as deformations and color transformations and evaluate how these additional sample augmentations affect the final accuracy. Generator. The role of the generator G, is to learn which transformations to input images are the most useful to train the classifier. In our intuition, learning an image transformation instead of learning a mapping from noise to image to generate additional samples, is an easier task in low data regime. Given an input image x i and a noise vector z, the generator G, composed of an encoder E N C, a decoder D EC and an affine transformer T, learns a transformation of the image that helps to train the classifier C. So, the transformation can be formulated as: DISPLAYFORM0 The loss of the generator can be formulated as: DISPLAYFORM1 where x i is an input image with class label y i, G(x i, z) is a transformation of the sample x i and a noise vector z. D C, D D and C are respectively the class discriminator, the dissimilarity discriminator and the classifier and will be addressed in the following paragraphs. α, β and γ are hyper-parameters introduced to balance the three loss terms and stabilize the training of the model. In the first term of the loss function, the probability that a pair (transformed sample, true label) is classified as fake is minimized. In the second term, the dissimilarity between the original image and the transformed image is maximized. Finally, in the third one, the predicted probability of the real label for the transformed sample is minimized in order to make the classifier robust against adversarial samples. Class discriminator. During training, the generator is supported by two discriminators. The first one is referred to as class discriminator D C. It ensures that the generated image belongs to the same class as the original image. D C takes as input an image and a class label and outputs the probability of the image to belong to that class. Its loss function is: DISPLAYFORM2 The first term increases the probability of D C for a real sample x i of class y i, whereas the second term reduces D C for a generated sample G(x i, z) of the same class. In this way the discriminator learns to distinguish between real and generated samples of a certain class. Dissimilarity discriminator. The second discriminator, called dissimilarity discriminator D D, ensures that the generated sample is as different as possible from the original sample. D D takes a pair of samples as input, and outputs a dissimilarity score between the two samples, ranging between 0 and 1, where 0 means that the two samples are identical. Its loss function can be formulated as: DISPLAYFORM3 In the first term of the loss function, the dissimilarity between the original sample and another true sample from the same class is maximized, whereas in the second one, the dissimilarity between a true sample and the corresponding transformed sample is minimized. Classifier. The image classifier C is trained jointly with the generator and the two discriminators. C is fed with real samples as well as augmented samples, i.e. samples transformed by G. Its loss function is: DISPLAYFORM4 In the first term of the loss function, the cross entropy loss between the predicted labels of the true samples and the true label distribution is minimized. In contrast, the cross entropy loss between the predicted labels of the transformed samples and the true label distribution is minimized in the second term. Global Loss. Finally, we want to minimize a global loss to find the optimal parameters for the generator, discriminators and classifier. This loss is defined as: DISPLAYFORM5 During optimization, we sequentially minimize a mini-batch of each loss. Notice that L G tries to minimize the cross-entropy of D C of the transformed samples G(x i, z), while L D C tries to minimize 1 − D C. The same also for D D and C. This is not a problem, in fact this shows that the defined loss is adversarial, in the sense that generator and discriminator//classifier fight to push the losses in different directions. If the optimization is tuned properly, this mechanism generates augmented samples that are good for training the classifier, i.e. samples that belongs to the right class but are close to the decision boundaries. In all our experiments, we apply a basic pre-processing to the images, which consists in subtracting the mean pixel value, and then dividing by the pixel standard deviation. The generator is a combination of a STN BID7 ) module followed by a U-Net BID11 ) network. The generator network takes as input an image and a Gaussian noise vector (100 dimensions), which are concatenated in the first layer of the network. The three parameters α, β and γ of the generator loss are estimated on a validation set. For the class discriminator D C, we use the same architecture as in BID4. The network is adapted to take as input an image and a label (as a one hot vector). These are concatenated and given as input to the first layer of the architecture. For the dissimilarity discriminator D D, we also use the same architecture. The network is adapted to take as input a pair of images, which are concatenated in the first layer of the architecture. For the classifier, we use the architecture used in BID4. We use Adam as optimization method. Training parameters To train our model, we use following values for the optimization parameters. Generator: Adam optimizer with a initial learning rate of 0.0005, a β 1 value of 0.5 and a β 2 value of 0.999. Class Discriminator: Adam optimizer with a initial learning rate of 0.0005, a β 1 value of 0.5 and a β 2 value of 0.999. As balance factor (see Sec. 2), we use as value for α 0.1 for MNIST, 1 for SVHN and 0.1 for CIFAR10. Similarity Discriminator: Adam optimizer with a initial learning rate of 0.0005, a β 1 value of 0.5 and a β 2 value of 0.999. As balance factor (see Sec. 2), we use as value for β 0.05 for MNIST, 1 for SVHN and 0.05 for CIFAR10. Classifier: Adam optimizer with a initial learning rate of 0.006, a β 1 value of 0.5 and a β 2 value of 0.999. As balance factor (see Sec. 2), we use as value for γ 0.005 for MNIST, 0.0005 for SVHN and 0.001 for CIFAR10.Detailed architectures In Tab. 3 we show the details of the classifier C, in Tab. 4 and Tab. 5, the details of respectively the two discriminators D C and D D. In Tab. 6, we can see the details for the generator G. Our approach learns to apply the right transformations for each dataset. For instance on MNIST and Fashion-MNIST there is no flip, nor zoom, because not useful, while on SVHN zoom is often used and on CIFAR10, both zoom, flip and color changes are applied.

Represent the TLDR for finding the corresponding scientific article.

Automatic Learning of data augmentation using a GAN based architecture to improve an image classifier

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.


Question: Represent the scientific article to find a short TLDR

Question: We consider the problem of information compression from high dimensional data. Where many studies consider the problem of compression by non-invertible trans- formations, we emphasize the importance of invertible compression. We introduce new class of likelihood-based auto encoders with pseudo bijective architecture, which we call Pseudo Invertible Encoders. We provide the theoretical explanation of their principles. We evaluate Gaussian Pseudo Invertible Encoder on MNIST, where our model outperform WAE and VAE in sharpness of the generated images. We consider the problem of information compression from high dimensional data. Where many studies consider the problem of compression by non-invertible transformations, we emphasize the importance of invertible compression as there are many cases where one cannot or will not decide a priori what part of the information is important and what part is not. Compression of images for person ID in a small company requires less resolution then person ID at an airport. To loose part of the information without harm to the future purpose of viewing the picture requires knowing the purpose upfront. Therefore, the fundamental advantage of invertible information compression is that compression can be undone if a future purpose so requires. Recent advances of classification models have demonstrated that deep learning architectures of proper design do not lead to information loss while still being able to achieve state-of-the-art in classification performance. These i-RevNet models BID5 implement a small but essential modification of the popular RevNet models while achieving invertibility and a performance similar to the standard RevNet BID2. This is of great interest as it contradicts the intuition that information loss is essential to achieve good performance in classification BID13. Despite the requirement of the invertibility, flow-based generating models BID0; BID11; BID6 demonstrate that the combination of bijective mappings allows one to transform the raw distribution of the input data to any desired distribution and perform the manipulation of the data. On the other hand, Auto-Encoders have provided the ideal mechanism to reduce the data to the bare minimum while retaining all essential information for a specific task, the one implemented in the loss function. Variational Auto Encoders (VAE) BID7 and Wasserstein Auto Encoders (WAE) BID14 are performing best. They provide an approach for stable training of autoencoders, which demonstrate good at reconstruction and generation. However, both of these methods involve the optimization of the objective defined on the pixel level. We would emphasise the importance of avoiding the separate decoder part and training the model without relying on the reconstuction quality directly. Combining the best of Invertible mappings and Auto-Encoders, we introduce Pseudo Invertible Encoder. Our model combines bijectives with restriction and extension of the mappings to the dependent sub-manifolds FIG0. The main contributions of this paper are the following:• We introduce new class of likelihood-based Auto-Encoders, which we call Pseudo Invertible Encoders. We provide the theoretical explanation of their principles.• We demonstrate the properties of Gaussian Pseudo Invertible Encoder in manifold learning.• We compare our model with WAE and VAE on MNIST, and report that the sharpness of the images, generated by our models is better. 2 RELATED WORK ResNets BID3 enable Networks to grow even more and thus memory consumption becomes a bottleneck. BID2 propose a Reversible Residual Network (RevNet) where each layer's activations can be reconstructed from the activations of the next layer. By replacing the residual blocks with coupling layers, they mimic the behaviour of residual blocks while being able to retrieve the original input of the layer. RevNet replaces the residual blocks of ResNets, but also accommodates non-invertible components to train more efficiently. By adding a downsampling operator to the coupling layer, i-RevNet circumvents these non-invertible modules BID5. With this they show that losing information is not a necessary condition to learn representations that generalize well on complicated problems. Although i-RevNet circumvents non-invertible modules, data is not compressed and the model is only invertible up to the last layer. All their methods do not allow dimensionality reduction. In current research we build a pseudo invertible model which performs dimensionality reduction. Auto-Encoders were first introduced by BID12 as an unsupervised learning algorithm. They are now widely used as a technique for dimension reduction by compressing input data. By training an encoder and a decoder network, and measuring the distance between original and reconstructed data, data can be represented in a latent space. This latent space can then be used for supervised learning algorithms. Instead of learning a compressed representation of the input data BID7 propose to learn the parameters of a probability distribution that represent the data. tol introduced new class of models -Wasserstein Auto Encoders, which use Optimal Transport to be trained. These methods require the optimization of the objective function which includes the terms defined on pixel level. Our model does not require such optimization. Moreover, it only perform encoding at training time. Here we introduce the approach for obtaining dimensionality reduction invertible mappings. Our method is based on the restriction of the mappings to low-dimensional manifolds, and extension of the inverse mappings with certain constraints (Fig. 2). Given data DISPLAYFORM0 In other words, we are looking for a pair of Figure 2: The schematic representation of the Restriction-Extension approach. The invertible mapping X ↔ Z is preformed by using the dependent sub-manifold R = g(Z) and a pair extended functionsG,G −1.associated functions G and G −1 such that DISPLAYFORM1 We use this residual manifold in order to match the dimensionalities of the hidden and initial spaces. Here we introduce the function g: DISPLAYFORM2 With no loss of generality we can say that R = g(Z). We use the pair of extended functionsG: DISPLAYFORM3 Rather than searching for the invertible dimensionality reduction mapping directly, we seek to find G, the invertible transformation with certain constraints, expressed by R.In search forG, we focus on DISPLAYFORM4 where F is a parametric family of functions invertible on R D. We select the function F θ with parameters θ which satisfy the constraint: DISPLAYFORM5 where DISPLAYFORM6 Taking into account constraint 3, we derive F θ (x) = [z, r], where z ∈ Z and r ∈ R. By combining this with Eq. 2 we have the desired pair of functions: DISPLAYFORM7 The obtained function G is Pseudo Invertible Endocer, or shortly PIE. As we are interested in high dimensional data such as images, the explicit choice of parameters θ is impossible. We choose θ * as a maximizer of the log likelihood of the observed data given the prior p θ (x): DISPLAYFORM0 After a change of variables according to Eq. 4 we obtain Taking into account the constraint 3 we derive the joint distribution for DISPLAYFORM1 DISPLAYFORM2 Dirac's delta function can be viewed as a limit of sequence of Gaussians: DISPLAYFORM3 Let us fix 2 = 2 0 DISPLAYFORM4 Finally, for the log likelihood we have: DISPLAYFORM5 We choose prior distribution p(z) as Standard Gaussian. We search for the parameters by using Gradient Descent. The method relies on the function F θ. This choice is challenging by itself. The currently known classes of real-value bijectives are limited. To overcome this issue, we approximate F θ with a composition of basic bijectives from certain classes: DISPLAYFORM0 where DISPLAYFORM1 Taking into account that a composition of PIE is also PIE, we create a final dimensionality reduction mapping from a sequence of PIEs: such that DISPLAYFORM2 DISPLAYFORM3 where DISPLAYFORM4 Then the log likelihood is represented as DISPLAYFORM5 where J kl is the Jacobian of the k-th function of the l-th PIE. The approximation error here depends only on, according to the Eq. 10. For the simplicity we will now refer to the whole model as PIE. The building blocks of this model are PIE blocks. If we choose the distribution p(z) in Eq. 17 as Standard Gaussian, g l (·) = 0, ∀l and 0 = 1, then the model can be viewed as Normalizing Flow with multi-scale architecture BID1 FIG1. It was demonstrated in BID1 that the model with such architecture achieves semantic compression. This section introduces the basic bijectives for the Pseudo-Invertible Encoder (PIE). We explain what each building bijective consists of and how it fits in the global architecture as shown in FIG2. PIE is composed of a series of convolutional blocks followed by linear blocks, as depicted in FIG2. The convolutional PIE blocks consist of series of coupling layers and 1×1 convolutions. We perform invertible downsampling of the image at the beginning of the convolutional block, by reducing the spatial resolution and increasing the number of channels, keeping the overall number of the variables the same. At the end of the convolutional PIE block, the split of variables is performed. One part of the variables is projected to the residual manifold R while others is feed to the next block. The linear PIE blocks are constructed in the same manner. However, the downsampling is not performed and 1 × 1 convolutions are replaced invertible linear mappings. DISPLAYFORM0 Figure 5: Structure of a coupling block. P partitions the input into two groups of equal length. U unites these group together. In the inverse P −1 and U −1 are the reverse of these operations respectively. In order to enhance the flexibility of the model, we utilize affine coupling layers Fig. 5. We modify the version, introduced in BID1.Given input data x, the output y is obtained by using the mapping: DISPLAYFORM0 Here multiplication and division are performed element-wise. The scalings s 1, s 2 and the biases b 1, b 2 are the functions, parametrized with neural networks. The invertibility is not required for this functions. x 1, x 2 are the non-intersecting partitions of x. For convolutional blocks we partition the tensors by splitting them into halves along the channels. In case of the linear blocks, we just split the features into halves. The log determinant of the Jacobian of coupling layer is given by: DISPLAYFORM1 where log | · | is calculated element-wise. The affine couplings operate on non-intersecting parts of the tensor. In order to capture the various correlations between channels and features, the different mechanism of channel permutations were proposed. BID6 demonstrated that invertible 1×1 convolutions perform better than fixed permutations and reversing of the order of channels BID1.We parametrize Invertible 1 × 1 Convolutions and invertible linear mappings with. Given the vector v, the Householder Matrix is computed as: DISPLAYFORM0 (b) Inverse Figure 6: Structure of the split method. P partitions the input into two sub samples. P −1 unites these sub samples together. The obtained matrix is orthogonal. Therefore, its inverse is just its transpose, which makes the computation of the inverse easier comparing to BID6. The log determinant of the Jacobian of such transformation is equal to 0. We use invertible downsampling to progressively reduce the spatial size of the tensor and increase the number of its channels. The downsampling with the checkerboard patterns Jacobsen et al. FORMULA3; BID1 transforms the tensor of size C ×H ×W into a tensor of size 4C × DISPLAYFORM0 where H, W are the height and the width of the image, and C is the number of the channels. The log determinant of the Jacobian of Downsampling is 0 as it just performs permutation. All the discussed blocks transform the data while preserving its dimensionality. Here we introduce Split block Fig. 6, which is responsible for the projection, restrictions and extension, described in Section 3. It reduces the dimensionality of the data by splitting the variables into two nonintersecting parts z, r of dimensionalities d and D − d, respectively. z is kept and is to be processed by the subsequent blocks. r is constrained to match N (r|g(z), 2 0 I). The mappings is defined as DISPLAYFORM0 For this experiment we trained a Gaussian PIE on the MNIST digits dataset. We build PIE with 2 convolutional blocks, each splitting the data in the last layer to 50% of the input size. Next we add three linear blocks to PIE, reducing the dimensions to 64, 10 and the last block does not reduce the dimensions any further. For each affine transformation we use the three biggest possible Householder reflections. For this experiment we set K l equal to 3. Optimization is done with the Adam optimizer BID8. The model diminishes the number of dimensions from R 784 to R 10.This experiment shows the ability of PIE to learn a manifold with three different constraints; 2 = 0.01, 2 = 0.1 and 2 = 1.0. The are shown in FIG3. As the constraint gets to loose, as shown in the right column, the model is not able to reconstruct anymore FIG3. Lower values for 2 perform better in terms of reconstruction. Too low values, however, sample fuzzy images FIG3. Narrowing down the distribution to sample from increases the models probability to produce accurate images. This is shown in FIG3 where samples are taken from N (0, 0.5). For both 2 = 0.01 and 2 = 0.1 reconstructed images are more accurate. This experiment shows that tightening the constraint by decreasing 2 increases the power of the manifold learned by the model. This is shown again in FIG3 where we diminished the number of dimensions even further from R 10 to R 2 utilizing UMAP BID9. With 2 = 1.0 UMAP created a manifold with a good Gaussian distribution. However, from the manifold created by PIE it was not able to separate distinct digits from each other. Tightening the constraint with a lower 2 moves the manifold created by UMAP further away from a Gaussian distribution, while it is better able to separate classes from each other. It is a well-known problem in VAEs that generated images are smoothened. improves over VAEs by utilizing Wasserstein distance function. To test the sharpness of generated images we convolve the grey-scaled images with the Laplace filter. This filter acts as an edge detector. We compute the variance of the activations and average them over 10000 sampled images. If an image is blurry, it means there are less edges and thus more activations will be close to zero, leading to a smaller variance. In this experiment we compare the sharpness of images generated by PIE with WAE, VAE and the sharpeness of the original images. For VAE and WAE we take the architecture as described in BID10. For PIE we take the architecture as described in section 5.1. Table 1 shows the for this experiment. PIE outperforms both VAE and WAE in terms of sharpeness of generated images. Images generated by PIE are even more sharp then original images from the MNIST dataset. An explanation for this is the use of a checkerboard pattern in the downsampling layer of the PIE convolutional block. With this technique we capture intrinsic properties of the data and are thus able to reconstruct sharper images. In this paper we have proposed the new class of Auto Encoders, which we call Pseudo Invertible Encoder. We provided a theory which bridges the gap between Auto Encoders and Normalizing Flows. The experiments demonstrate that the proposed model learns the manifold structure and generates sharp images.

Represent the TLDR for finding the corresponding scientific article!

New Class of Autoencoders with pseudo invertible architecture

Represent the short summary for finding the corresponding scientific article (long)

This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.