Question: Represent the scientific article to find a short TLDR

Question: One of the unresolved questions in deep learning is the nature of the solutions that are being discovered. We investigate the collection of solutions reached by the same network architecture, with different random initialization of weights and random mini-batches. These solutions are shown to be rather similar - more often than not, each train and test example is either classified correctly by all the networks, or by none at all. Surprisingly, all the network instances seem to share the same learning dynamics, whereby initially the same train and test examples are correctly recognized by the learned model, followed by other examples which are learned in roughly the same order. When extending the investigation to heterogeneous collections of neural network architectures, once again examples are seen to be learned in the same order irrespective of architecture, although the more powerful architecture may continue to learn and thus achieve higher accuracy. This pattern of remains true even when the composition of classes in the test set is unrelated to the train set, for example, when using out of sample natural images or even artificial images. To show the robustness of these phenomena we provide an extensive summary of our empirical study, which includes hundreds of graphs describing tens of thousands of networks with varying NN architectures, hyper-parameters and domains. We also discuss cases where this pattern of similarity breaks down, which show that the reported similarity is not an artifact of optimization by gradient descent. Rather, the observed pattern of similarity is characteristic of learning complex problems with big networks. Finally, we show that this pattern of similarity seems to be strongly correlated with effective generalization. The recent success of deep networks in solving a variety of classification problems effectively, in some cases reaching human-level precision, is not well understood. One baffling is the incredible robustness of the learned models: using variants of Stochastic Gradient Descent (SGD), with random weight initialization and random sampling of mini-batches, different solutions are obtained. While these solutions typically correspond to different parameter values and possibly different local minima of the loss function, nevertheless they demonstrate similar performance reliably. To advance our understating of this issue, we are required to compare different network instances. Most comparison approaches (briefly reviewed in Appendix A) are based on deciphering the internal representations of the learned models (see ; ; ; ;). We propose a simpler and more direct approachcomparing networks by their classifications of the data. To this end, we represent each network instance by 2 binary vectors which capture the train and test classification accuracy. Each vector's dimension corresponds to the size of the train/test dataset; each element is assigned 1 if the network classifies the corresponding data point correctly, and 0 otherwise. Recall the aforementioned empirical observation -different neural network instances, obtained by repeatedly training the same architecture with SGD while randomly sampling its initial weights, achieve similar accuracy. At the very least, this observation predicts that the test-based vector representation of different networks should have similar L 1 /L 2 norms. But there is more: it has been recently shown that features of deep networks capture perceptual similarity reliably and consistently, similarly across different instances and different architectures . These seem to suggest that our proposed representation vectors may not only have a similar norm, but should also be quite similar as individual vectors. But similar in what way? In this paper, we analyze collections of deep neural networks classifiers, where the only constraint is that the instances are trained on the same classification problem, and investigate the similarity between them. Using the representation discussed above, we measure this similarity by two scores, consistency score and consensus score, as defined in §2. Like other comparison approaches (see Appendix A), our analysis reveals a high level of similarity between trained networks. Interestingly, it reveals a stronger sense of similarity than previously appreciated: not only is the accuracy of all the networks in the collection similar, but so is the pattern of classification. Specifically, at each time point during the learning process (or in each epoch), most of the data points in both the train and test sets are either classified correctly by all the networks, or by none at all. As shown in §3, these are independent of choices such as optimization method, hyperparameter values, the detailed architecture, or the particular dataset. They can be replicated for a fixed test set even when each instance in the collection sees a different train set, as long as the training data is sampled from the same distribution. Moreover, the same pattern of similarity is observed for a wide range of test data, including out-of-sample images of new classes, randomly generated images, or even artificial images generated by StyleGAN . These are also reproduce-able across domains, and were reproduced using BiLSTM with attention for text classification. We may therefore conclude that different network instances compute similar classification functions, even when being trained with different training samples. It is in the dynamic of learning, where the of our analysis seem to go significantly beyond what has been shown before, revealing an even more intriguing pattern of similarity between trained NN instances. Since deep NNs are almost always trained using gradient descent, each network can be represented by a time series of train-based and test-based representation vectors, one per epoch. We find that network instances in the collection do not only show the same pattern of classification at the end of the training, but they also evolve in the same way across time and epochs, gradually learning to correctly or incorrectly classify the same examples in the same order. When considering bigger classification problems such as the classification of ImageNet with big modern CNN architectures, a more intricate pattern of dynamics is evident: to begin with, all networks wrongly classify most of the examples, and correctly classify a minority of the examples. The learning process is revealed by examples moving from one end (100% false classification) to the other end (100% correct classification), which implies two things: (i) the networks learn to correctly classify examples in the same order; (ii) the networks agree on the examples they misclassify throughout. As shown in §4, these hold regardless of the network' architecture. To drive this point home we compare a variety of public domain architectures such as VGG19 , AlexNet , DenseNet and ResNet-50 . In all cases, different architectures may learn at a different pace and achieve different generalization accuracy, but they still learn in the same order. Thus all networks start by learning roughly the same examples, but the more powerful networks may continue to learn additional examples as learning proceeds. A related phenomenon is observed when extending the analysis to simpler learning paradigms, such as deep linear networks, SVM, and KNN classifiers. Our empirical study extends to cases where these robust patterns of similarity break down, see §5. For example, when randomly shuffling the labels in a known benchmark , the agreement between different classifiers disappear. This stands in agreement with , where it is shown that networks that generalize are more similar than those that memorize. Nevertheless, the similarity in learning dynamic is not an artifact of learnability, or the fact that the networks have converged to solutions with similar accuracy. To see this we constructed a test case where shallow CNNs are trained to discriminate an artificial dataset of images of Gabor patches (see Appendix C). Here it is no longer true that different network instances learn in the same order; rather, each network instance follows its own path while converging to the final model. The similarity in learning dynamic is likewise not an artifact of using gradient descent. To see this we use SGD to train linear classifiers to discriminate vectors sampled from two largely overlapping Gaussian distributions. Once again, each classifier follows its own path while converging to the same optimal solution. Given some neural network architecture f, and a labeled dataset denotes a single data point and y i ∈ [K] its corresponding label, we define and analyze the consistency of f when repeatedly trained on X from scratch. Let S E denote the set of different extents (total epochs of X) used to train f, where |S E | = E. ∀e ∈ S E we create a collection of N instances of f, denoted F e = {f e 1, ..., f e N}. Each instance f e i is initialized independently using Xavier initialization , then trained with SGD on randomly sampled mini-batches for e epochs. We measure the consistency of architecture f by comparing the predictions of the different instances in each collection F e, and analyze this consistency throughout the entire learning process as it changes with e. Thus for epoch e, we define the consistency score of an example (x, y) as follows: The consistency score c e (x, y) measures the classifiers' agreement when x is correctly classified. However, it does not take into account the classifiers' agreement when it is not. We therefore define in addition the consensus score, a complementary score that measures the consensus of the classifiers -the largest number of classifiers that classify each example by the same label: We say that example e (x j, y j). We say that example (x i, y i) is learned at least as fast as example (x j, y j) if from some epoch e and onward, example (x i, y i) is easier than example (x j, y j). This is formally given by: ∃e ∈ S E: ∀e > e c e (x i, y i) ≥ c e (x j, y j). When all the classifiers in F e are identical, the consistency score of each example is either 0 or 1, and its consensus is 1. This in a perfect bi-modal distribution of consistency scores; we quantify bi-modality using the following measure suggested by Pearson for an RV X: kurtusis(X) − skewness 2 (X) − 1; the lower this measure is, the more bi-modal X is. On the other hand, when all classifiers independently choose class labels, the distribution of both scores is expected to resemble a Gaussian 1, centered around the average accuracy of the classifiers for the consistency score, and a slightly higher value for the mean consensus score. It follows that the higher the mean consensus is, and the more bi-modal-like the distribution of consistency scores is around 0 and 1, the more similar the set of classifiers is. Throughout this paper we empirically investigate the distribution of the two scores defined above, using different architectures and datasets. We also examine the effects of other factors, such as resampling the train data, or classifying out-of-sample test data. For the most part, we focus on CNNs trained on visual classification tasks (although the are reproduced in other domains), and analyze the distribution of consistency scores throughout the entire learning process. In this section, we investigate collections of classifiers obtained from a single NN architecture f. Same training set. We start with the simplest condition, where all instances in collection F are obtained by training with the same training set X, with different initial conditions and with independently sampled mini-batches. When using datasets of natural images, during learning the consistency among all networks in F is high, see Figs. 1a, b and Fig. 2. Upon initialization, all networks are effectively i.i.d random variables, and therefore the distribution of consistency scores is approximately normal around random chance 1 -1 K for K labels. After a few epochs (in many cases a single epoch is enough, see Appendix D), the consistency distribution changes dramatically, transforming to a bi-modal distribution peaking around 0 and 1. This abrupt distribution change is robust, and rather striking: from a state where most of the examples are being classified correctly by 1 K of the networks, now most of the examples are being misclassified by all the networks, while a small fraction is being correctly classified by all the networks. When learning proceeds, the improvement in accuracy affects a shift of points from the peak at 0 to the peak at 1, and the distribution remains bi-modal, see Figs. 1a,b and Fig. 2. The data is learned in a specific order which is insensitive to the network initialization and the sampling of the mini-batches. This is true for both the train and test sets. It indicates that the networks capture similar functions in corresponding epochs: they classify correctly the same examples, and also consistently misclassify the same examples. Had the learning of the different network instances progressed independently, we would have seen a different dynamic: as illustrated in Fig. 1c, the distribution of consistency scores of independent learners remains Gaussian in all epochs, where the Gaussian's mean slowly shifts while tracking the improved accuracy. The consistency score is not affected by how similar the misclassifications are. For this purpose we have the consensus score, which measures consistency regardless of whether the label is true or false: a consensus score of 1 indicates that all networks have classified the datapoint in the same way, regardless of whether it is correct or incorrect. Fig. 3a shows the distribution of consensus scores for the cases depicted in Fig. 1, showing that indeed all the network instances classify examples in almost the same way, even when they misclassify. Had the learning of the different network instances progressed independently, the dynamic would have been different as shown in We investigated a variety of public domain architectures, including AlexNet, DenseNet and ResNet-50 for ImageNet, VGG19 and a stripped version of VGG (denoted st-VGG) for CIFAR-10 and CIFAR-100, and several different handcrafted networks for other data sets (see details in Appendix B). The can be replicated when changing the following hyper-parameters: learning rate, optimizer, batch size, dropout, L2-regularization, width, length and depth of layers, number of layers, number and size of kernels, and activation functions. These hyper-parameters differ across the experiments detailed both in the main paper and in Appendix D. We now extend the analysis of the previous section to include networks instances which are generated by different architectures. In §4.3 we discuss comparisons with other classifiers. We start by directly extending the previous analysis to two collections generated by two different architectures. Each architecture is characterized by its own learning pace, therefore it makes little sense to compare consistency epoch-wise. Instead, we first match epochs between the two collections: in a matched pair of epochs [e 1, e 2], by definition the mean error of the first collection in epoch e 1 is equivalent to the mean error of the second collection in epoch e 2 4. For each pair of epochs, we merge the corresponding collections of the two architectures and compute the consistency of the merged collection. We call this score cross-consistency. In The distribution of the cross-consistency score in Fig. 6a,b implies that the two architectures, ResNet-50 and AlexNet, learn the data in the same order when trained on the same dataset. We wish to measure this implication directly. To this end, we measure for a given point and a given collection, how fast the point has been learned by all the instances in the collection. Note that the sooner a point is being learned by all network instances, the higher its average consistency should be when computed over all epochs. We therefore call a point's average consistency its accessibility score, and correlate this score across two collections to compare the order of learning. We start by correlating the accessibility score of two collections generated by the same architecture. When comparing two collections of ResNet-50, the correlation is almost 1 (r = 0.99, p ≤ 10 (Fig. 25). Convolutional Neural Networks where the internal operations are limited to linear operators define an important class of CNNs, as their linearity is often exploited in the theoretical investigation of deep learning. It is natural to wonder, therefore, whether the bi-modal behavior observed in general CNNs also occurs in this case. The answer is in the affirmative. We train 100 st-VGG networks on the small-mammals dataset (see Appendix C). By replacing all the activation layers by the identity operator, and changing the max-pooling into average-pooling, a linear CNN architecture is obtained. The performance of these linear networks is weaker (0.43 average accuracy) than the original non-linear networks (0.56 average accuracy). Still, the distribution of the consistency scores throughout the entire learning process is bi-modal (maximum Pearson: 0.055), and this bi-modality is even more pronounced than the bi-modality in the non-linear case (maximum Pearson: 0.22). The bi-modal dynamics of st-VGG can be seen in the top row of Linear networks converge in just a few epochs, hence ruling the cross-consistency score uninformative in this case. Nevertheless, we still observe that linear and non-linear networks learn examples in roughly the same order, as shown in Fig. 7b. To understand Fig. 7b, we define for each epoch the set of "easiest" points, including all the points whose corresponding consistency score is larger than 0.9. For each epoch, there are two such sets: C Up to now, we investigated a variety of neural network architectures, revealing a common learning pattern. Can we see the same commonalities with other classification methods and paradigms? First, we consider boosting based on linear classifiers as weak learners, because the training of both neural networks and AdaBoost share a dynamic aspect: in neural networks training accuracy increases with time due to the use of GD, while in AdaBoost accuracy increases over time due to the accumulation of weak learners. We find that in both dynamics, there are commonalities in the learning order of examples. Next, we consider other machine learning paradigms, including SVM, KNN classifier, perceptron, decision tree, random forest and Gaussian naïve Bayes. Interestingly, we still find a strong correlation with the order of learning as defined above, in that these classifiers tend to fit those examples which the neural networks learn first. These are discussed in full in Appendix F. In Section 3 we discussed the characteristic bi-modal distribution of consistency scores, illustrated in Fig. 1, which has appeared in all the experiments presented until now, in both the train and test sets. We have already seen that this bi-modality weakens as the similarity between the distributions of train and test data is reduced (see Fig. 5). In this section, we investigate the circumstances under which the bi-modal distribution of consistency scores is no longer seen. Learning to see Gabor patches. The bi-modal distribution of consistency scores through all stages of learning is not an inherent property of neural networks. We demonstrate this point using a dataset of artificial images, consisting of Gabor patches: the dataset contains 12 overlapping classes which differ from each other in orientation, parity and color (see Appendix C). 100 instances of st-VGG have been trained to classify this data. Now the distribution of consistency scores, shown in Fig. 8, is no longer bi-modal. Rather, the distribution is approximately normal most of the time. As learning proceeds, the mean of the distribution slowly shifts towards 1, and the width of the distribution seems to expand. At convergence, the models have reached similar performance, and the bi-modal characteristics partially re-appears on the test data. Random labels. Bi-modality seems to be associated with successful generalization. To see this, we take the small-mammals dataset, and reshuffle the labels such that every image is assigned a random label (following). In this case, training accuracy can reach 100% when disabling dropout (that acts as a regularizer), which indicates that the networks can memorize the data. Interestingly, the distribution of consistency scores in no longer bi-modal, with minimum Pearson score of 1.07 on train set and 1.35 on the test set during the entire learning process. Rather, the distribution in each epoch resembles a Gaussian centered around the mean accuracy of the networks, see Fully connected networks. Bi-modality is not an artifact of using gradient descent for optimization. This can be seen in the following analysis. Specifically, we consider a fully connected neural network architecture, with 2 intermediate layers, ELU and dropout. The networks are trained to discriminate points sampled from two largely overlapping Gaussian distributions in high dimension. The dynamic distribution of consistency scores is shown in Fig. 10, and resembles the distribution of independent networks shown in Fig. 1c. While the final solutions of the networks are similar, the order in which examples are being learned when employing SGD optimization is different. We empirically show that neural networks learn similar classification functions. More surprisingly with respect to earlier work, the learning dynamics is also similar, as they seem to learn similar functions also in intermediate stages of learning, before convergence. This is true for a variety of architectures, including different CNN architectures and LSTMs, irrespective of size and other hyper-parameters of the learning algorithms. We have verified this pattern of using many different CNN architectures, including most of those readily available in the public domain, and many of the datasets of natural images which are in common use when evaluating deep learning. The similarity of network instances is measured in the way they classify examples, including known (train) and new examples. Typically, the similarity over test data is as pronounced as it is over train data, as long as the train and test examples are sampled from the same distribution. We show that this similarity extends also to out of sample test data, but it seems to decrease as the gap between the distribution of the train data and the test data is increased. This pattern of similarity crosses architectural borders: while different architectures may learn at a different speed, the data is learned in the same order. Thus all architectures which reach a certain error rate seem to classify, for the most part, the same examples in the same manner. We also see that stronger architectures, which reach a lower generalization error, seem to start by first learning the examples that weaker architectures classify correctly, followed by the learning of some more difficult examples. This may suggest that the order in which data is learned is an internal property of the data. We also discuss cases where this similarity breaks down, indicating that the observed similarity is not an artifact of using stochastic gradient descent. Rather, the observed pattern of similarity seems to characterize the learning of complex problems with big networks. Curiously, the deeper the network is and the more non-linearities it has, and even though the model has more learning parameters, the progress of learning in different network instances becomes more similar to each other. Un-intuitively, this suggests that in a sense the number of degrees of freedom in the learning process is reduced, and that there are fewer ways to learn the data. This effect seems to force different networks, as long they are deep enough, to learn the dataset in the same way. This counter-intuitive joins other non-intuitive , like the theoretical that a deeper linear neural network converges faster to the global optimum than a shallow network . We also show that the observed pattern of similarity is strongly correlated with effective generalization. What does it tell us about the generalization of neural networks, a question which is considered by many to be poorly understood? Neural networks can memorize an almost limitless number of examples, it would seem. To achieve generalization, most training protocols employ some regularization mechanism which does not allow for unlimited data memorization. As a , the network fits only the train and test examples it would normally learn first, which are, based on our analysis, also the "easier" (or more typical) examples. We hypothesize that this may explain why a regularized network discovers robust solutions, with little variability among its likely instances. , and they can learn any arbitrary complex function . This extended capacity can indeed be reached, and neural networks can memorize datasets with randomly assigned labels . Nevertheless, the dominant hypothesis today is that in natural datasets they "prefer" to learn an easier hypothesis that fits the data rather than memorize it all . Our work is consistent with a hypothesis which requires fewer assumptions, see Section 6. The direct comparison of neural representations is regarded to be a hard problem, due to a large number of parameters and the many underlying symmetries. Many non-direct approaches are available in the literature: compare subsets of similar features across multiple networks, which span similar low dimensional spaces, and show that while single neurons can vary drastically, some features are reliably learned across networks. proposed the SVCCA method, which can compare layers and networks efficiently, with an amalgamation of SVD and CCA. They showed that multiple instances of the same converged network are similar to each other and that networks converge in a bottom-up way, from earlier layers to deeper ones. builds off the of , further showing that networks which generalize are more similar than ones which memorize, and that similarity grows with the width of the network. In various machine learning methods such as curriculum learning , self-paced learning and active learning , examples are presented to the learner in a specific order . Although conceptually similar, here we analyze the order in which examples are learned, while the aforementioned methods seek ways to alter it. Likewise, the design of effective initialization methods is a striving research area (; ;). Here we do not seek to improve these methods, but rather analyze the properties of a collection of network instances generated by the same initialization methodology. In addition to the public domain architectures described in §3, we also experimented with some handcrafted networks. Such networks are simpler and faster to train, and are typically used to investigate the learning of less commonly used datasets, such as the small-mammals dataset and tiny ImageNet. Below we list all the architectures used in this paper. st-VGG. A stripped version of VGG which we used in many experiments. It is a convolutional neural network, containing 8 convolutional layers with 32, 32, 64, 64, 128, 128, 256, 256 filters respectively. The first 6 layers have filters of size 3 × 3, and the last 2 layers have filters of size 2 × 2. Every second layer there is followed by 2 × 2 max-pooling layer and a 0.25 dropout layer. After the convolutional layers, the units are flattened, and there is a fully-connected layer with 512 units followed by 0.5 dropout. When training with random labels, we removed both dropout layers to enable proper training, as suggested in. The batch size we used was 100. The output layer is a fully connected layer with output units matching the number of classes in the dataset, followed by a softmax layer. We trained the network using the SGD optimizer, with cross-entropy loss. When training st-VGG, we used a learning rate of 0.05 which decayed by a factor of 1.8 every 20 epochs. Small st-VGG. To compare st-VGG with another architecture, we created a smaller version of it: we used another convolutional neural network, containing 4 convolutional layers with 32, 32, 64, 64 filters respectively, with filters of size 3 × 3. Every second layer there is followed by 2 × 2 max-pooling and a 0.25 dropout layer. After the convolutional layers, the units are flattened, and there is a fully-connected layer with 128 units followed by 0.5 dropout. The output layer is a fully connected layer with output units matching the number of classes in the dataset, followed by a softmax layer. We trained the network using the SGD optimizer, with cross-entropy loss. We trained this network with the same learning rate and batch size as st-VGG. MNIST architecture. When experimenting with the MNIST dataset, we used some arbitrary small architecture for simplicity, as most architectures are able to reach over 0.99 accuracy. The architecture we used had 2 convolutional layers, with 32 and 64 filters respectively of size 3 × 3. After the convolutions, we used 2 × 2 max-pooling, followed by 0.25 dropout. Finally, we used a fully connected layer of size 128 followed by 0.5 dropout and Softmax. We used a learning rate of 1 for 12 epochs, using AdaDelta optimizer and a batch size of 100. Fully connected architecture. When experimenting with fully connected networks, we used a 4 layers network, which simply flattened the data, followed by 2 fully connected layers with 1024 units, followed byn an output layer with softmax. We used 0.5 dropout after each fully connected layer. Since these networks converge fast, a wide range of learning rates can be used. Specifically, we used 0.04. We experimented with a wide range of numbers of fully connected layers, reaching similar . BiLSTM with Attention. When experimenting on textual data we used a GloVe embeddings, a layer of BiLSTM of size 300, 0.25 dropout and recurrent dropout, an attention layer, a fully connected layer of size 256 with 0.25 dropout and a last fully connected layer to extract output. The networks were optimized using Adam optimization with a learning rate of 0.005 and a batch size of 256. Small Mammals. The small-mammals dataset used in the paper is the relevant super-class of the CIFAR-100 dataset. It contains 2500 train images divided into 5 classes equally, and 500 test images. Each image is of size 32 × 32 × 3. This dataset was chosen due to its small size, which allowed for efficient experimentation. All the observed in this dataset were reproduced on large, public domain datasets, such as CIFAR-100, CIFAR-10, and ImageNet. Insect. Similarly to the small mammals dataset, the relevant super-class of CIFAR-100. Fish. Similarly to the small mammals dataset, the relevant super-class of CIFAR-100. Cats and Dogs. The cats and dogs dataset is a subset of CIFAR-10. It uses only the 2 relevant classes, to create a binary problem. Each image is of size 32 × 32 × 3. The dataset is divided to 20000 train images (10000 per class) and 2000 test images (1000 per class). Gabor. The Gabor dataset used in the paper, is a dataset we created which contains 12 classes of Gabor patches. Each class contains 100 images of Gabor patches which vary in size and orientation. Classes differ from each other in 3 parameters: 1) Parity -there is a different class for odd and even Gabor patches (corresponding to the use of sine or cosine respectively). 2) RGB channel -each class depicts the Gabor patch in a single RGB channel. 3) Orientation -each class can have one of the following base orientations: 45 •. The orientation of each class varies by ±30 •, making some of the classes non-separable, while some classes are. Code for creating this dataset will be published upon acceptance. Gaussian. The Gaussian dataset used in the fully connected case, is a 2-classes dataset. One class is sampled from a multivariate Gaussian with mean 0 and Σ = I, while the other class is sampled from a multivariate Gaussian with mean 0.1 and Σ = I. Other choices for the mean and variance yield similar . Each sampled vector was of dimension 3072, and then reshaped to 32 × 32 × 3 to resemble the shape of CIFAR images. Each class contained 2500 train images and 500 test images. VGGFace2 subset. We created a classification task for face recognition, using a subset of 10 classes from VGGFace2. We chose the classes containing the largest number of images. We chose 600 images from each class arbitrarily to be the train set, while the remaining points (between 89 and 243) served as the test set. Each image was resized to 64 × 64 × 3, using center cropping while maintaining aspect ratio. Stack Overflow. The data from Stack Overflow is publicly shared and used for tutorials. It contains 39K training samples and 1K test samples, each tagged with one of 20 programming languages as the language the question asks about. Each question must be regarded more as a paragraph than a sentence. Many words, terms and symbols are expected to be domain-dependent, and therefore under-represented in the embeddings. Similar qualitative were obtained in all the experiments with natural datasets. To maintain a fair comparison across epochs, the for each shown epoch e (effectively epoch extent) were obtained by independently training a different set of N networks from scratch for e epochs. The specific set of epochs S E, where |S E | = 7, that was used in each plot was determined arbitrarily, to evenly span all sections of learning. All the networks in all test cases converged before the final epoch plotted. Induced class hierarchy. The ranking of training examples induced by the consistency scores typically induces a hierarchical structure over the different classes as well. To see this, we train 100 instances of st-VGG on the small-mammals dataset, and calculate for each image the most frequent class label assigned to it by the collection of networks. In Fig. 27 we plot the histogram of the consistency score (as in Fig. 1), but this time each image is assigned a color, which identifies its most frequent class label (1 of 5 colors). It can be readily seen that at the beginning of learning, only images from 2 classes reach a consistency score of 1. As learning proceeds, more class labels slowly emerge. This suggests that classes are learned in a specific order, across all networks. Moreover, we can see a pattern in the erroneous label assignments, which suggests that the classifiers initially use fewer class labels, and only become more specific later on in the learning process. Dynamics of individual image consistency. We now focus on consistency scores of individual images, as they evolve throughout the entire learning process. For most examples, the score may climb up from random (0.2) to 1 in 1 epoch, it may dip down to 0 and then go up to 1 after a few epochs, or it may go rapidly down to 0. Either, the score remains 1 or 0. These patterns are shown in Fig. 29, and support the bi-modality we report above. The duration in which a certain example maintains a consistency score 0 correlates with the order of learning: the longer it has 0 consistency, the more difficult it is. A minority of the training examples exhibit different patterns of learning. For example, a few images (the green curve in Fig. 29) begin with a high consistency score (near 1), but after a few epochs their score drops to 0 and remains there. and small st-VGG (N = 100), and finally randomly picked super-classes of CIFAR-100, specifically "aquatic-mammals", "insects" and "household furniture" with st-VGG (N = 100). The number of instances N is chosen according to our computational capabilities. However, in all cases, picking much smaller N suffice to yield the same qualitative . In addition to hyper-parameters which may differ between various architectures, we also experimented with changing the hyper-parameters of st-VGG trained on the small-mammals dataset, always observing the same qualitative . All experiments used N = 100 instances. Specifically, we tried a large range of learning rates, learning rate decay, SGD and Adam optimizers, large range of batch sizes, dropout and L2-regularization. Cross architectures diversity. In addition to the in section 4, the same qualitative were obtained for all 2 architectures we trained on the same unmodified dataset. We conducted the following experiments: ImageNet dataset: ResNet-50 vs DenseNet, AlexNet vs DenseNet. Aquatic-mammals and small-mammals super-classes of CIFAR-100: st-VGG vs small st-VGG, Tiny ImageNet: st-VGG vs small st-VGG, CIFAR-10 and CIFAR-100: VGG19 vs st-VGG. All of which yielding similar to the ones analyzed in section 4. Boosting linear classifiers. We use AdaBoost with k weak linear classifiers, trained on the small-mammals dataset. As commonly observed, adding more classifiers (increasing k) improves the final performance. In Fig. 30a we plot accuracy for 3 groups of examples -easy, intermediate and difficult, where grouping is based on the consistency score of the CNN described above. The accuracy of boosting over easy examples is significantly higher than the general accuracy and does not improve as we increase k. This suggests that most of the easy examples can be classified linearly, and are learned first by both boosting and our CNNs. On the other hand, accuracy when classifying difficult examples is worse than the general accuracy and slowly decreases with k. Other classifiers. We train several classifiers on the small-mammals dataset: SVM (accuracy: 0.48), KNN classifier (accuracy: 0.36), perceptron (accuracy: 0.35), decision tree (accuracy: 0.31), random forest (accuracy: 0.48) and Gaussian naïve Bayes classifier (accuracy: 0.38). All classifiers under-perform the CNN architecture described above (accuracy: 0.56). As in the case of boosting, for each method, most of the examples which are classified correctly are among the first to be learned by the CNN architecture. In other words, for all these methods, which achieve low accuracy, it is the easier examples which are being learned for the most part. This is illustrated in Fig. 31, demonstrating the phenomenon that during the learning process, most of the examples that are being learned by other paradigms are the ones already learned by neural networks.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Most neural networks approximate the same classification function, even across architectures, through all stages of learning.

Represent the short summary for finding the corresponding scientific article (long)

We introduce the NLC, a metric that is cheap to compute in the networks randomly initialized state and is highly predictive of generalization, at least in fully-connected networks.


Question: We propose a unified framework for building unsupervised representations of individual objects or entities (and their compositions), by associating with each object both a distributional as well as a point estimate (vector embedding). This is made possible by the use of optimal transport, which allows us to build these associated estimates while harnessing the underlying geometry of the ground space. Our method gives a novel perspective for building rich and powerful feature representations that simultaneously capture uncertainty (via a distributional estimate) and interpretability (with the optimal transport map). As a guiding example, we formulate unsupervised representations for text, in particular for sentence representation and entailment detection. Empirical show strong advantages gained through the proposed framework. This approach can be used for any unsupervised or supervised problem (on text or other modalities) with a co-occurrence structure, such as any sequence data. The key tools underlying the framework are Wasserstein distances and Wasserstein barycenters (and, hence the title!). One of the main driving factors behind the recent surge of interest and successes in natural language processing and machine learning has been the development of better representation methods for data modalities. Examples include continuous vector representations for language , convolutional neural network (CNN) based text representations (; ; ; BID4, or via other neural architectures such as RNNs, LSTMs BID14 Collobert and Weston, 1 And, hence the title! 2008), all sharing one core idea -to map input entities to dense vector embeddings lying in a lowdimensional latent space where the semantics of the inputs are preserved. While existing methods represent each entity of interest (e.g., a word) as a single point in space (e.g., its embedding vector), we here propose a fundamentally different approach. We represent each entity based on the histogram of contexts (cooccurring with it), with the contexts themselves being points in a suitable metric space. This allows us to cast the distance between histograms associated with the entities as an instance of the optimal transport problem (Monge, 1781; ;). For example, in the case of words as entities, the ing framework then intuitively seeks to minimize the cost of moving the set of contexts of a given word to the contexts of another. Note that the contexts here can be words, phrases, sentences, or general entities cooccurring with our objects to be represented, and these objects further could be any type of events extracted from sequence data, including e.g., products such as movies or web-advertisements BID8, nodes in a graph BID9, or other entities . Any co-occurrence structure will allow the construction of the histogram information, which is the crucial building block for our approach. A strong motivation for our proposed approach here comes from the domain of natural language, where the entities (words, phrases or sentences) generally have multiple semantics under which they are present. Hence, it is important that we consider representations that are able to effectively capture such inherent uncertainty and polysemy, and we will argue that histograms (or probability distributions) over embeddings allows to capture more of this information compared to point-wise embeddings alone. We will call the histogram as the distributional estimate of our object of interest, while we refer to the individual embeddings of single contexts as point estimates. Next, for the sake of clarity, we discuss the framework in the concrete use-case of text representations, when the contexts are just words, by employing the well-known Positive Pointwise Mutual Information (PPMI) matrix to compute the histogram information for each word. With the power of optimal transport, we show how this framework can be of significant use for a wide variety of important tasks in NLP, including word and sentence representations as well as hypernymy (entailment) detection, and can be readily employed on top of existing pre-trained embeddings for the contexts. The connection to optimal transport at the level of words and contexts paves the way to make better use of its vast toolkit (like Wasserstein distances, barycenters, etc.) for applications in NLP, which in the past has primarily been restricted to document distances (; BID16 .We demonstrate that building the required histograms comes at almost no additional cost, as the co-occurrence counts are obtained in a single pass over the corpus. Thanks to the entropic regularization introduced by , Optimal Transport distances can be computed efficiently in a parallel and batched manner on GPUs. Lastly, the obtained transport map FIG0) also provides for interpretability of the suggested framework. Most of the previous work in building representations for natural language has been focused towards vector space models, in particular, popularized through the groundbreaking work in Word2vec and GloVe . The key idea in these models has been to map words which are similar in meaning to nearby points in a latent space. Based on which, many works (a; ;) have suggested specializing the embeddings to capture some particular information required for the task at hand. One of the problems that still persists is the inability to capture, within just a point embedding, the various semantics and uncertainties associated with the occurrence of a particular word BID15.A recent line of work has proposed the view to represent words with Gaussian distributions or mixtures of Gaussian distributions (b;), or hyperbolic cones BID5 for this purpose. Also, a concurrent work from has suggested using elliptical distributions endowed with a Wasserstein metric. While these already provide richer information than typical vector embeddings, their form restricts what could be gained by allowing for arbitrary distributions. In addition, hyperbolic embeddings (; BID5 are so far restricted to supervised tasks (and even elliptical embeddings to a most extent), not allowing unsupervised representation learning as in the focus of the paper here. To this end, we propose to associate with each word a distributional and a point estimate. These two estimates together play an important role and enable us to make use of optimal transport. Amongst the few explorations of optimal transport in NLP, i.e., document distances (; BID16, document clustering , bilingual lexicon induction , or learning an orthogonal Procrustes mapping in Wasserstein distance BID7, the focus has been on transporting words directly. For example, the Word Mover's Distance casts finding the distance between documents as an optimal transport problem between their bag of words representation. Our approach is different as we consider the transport over contexts instead, and use it to propose a representation for words. This enables us to establish any kind of distance (even asymmetric) between words by defining a suitable underlying cost on the movement of contexts, as we show for the case of entailment. Another benefit of defining this transport over contexts is the added flexibility to extend the representation for sentences (or arbitrary length text) by utilizing the idea of Wasserstein barycenters, which to the best of our knowledge has never been considered in the past. Lastly, the proposed framework is not specific to words or sentences but holds for building unsupervised representations for any entity and composition of entities, where a co-occurrence structure can be devised between entites and their contexts. Optimal Transport (OT) provides a way to compare two probability distributions defined over a space G, given an underlying distance on this space (or more generally a cost of moving one point to another). In other terms, it lifts distance between points to distance between distributions. Below, we give a short yet formal description on optimal transport for the discrete case. Let's consider an empirical probability measure of the form µ = n i=1 a i δ(x i) where X = (x 1, . . ., x n) ∈ G n, δ(x) denotes the Dirac (unit mass) distribution at point x ∈ G, and (a 1, . . ., a n) lives in the probability simplex DISPLAYFORM0 If the ground cost of moving from point x i to y j is denoted by M ij, then the Optimal Transport distance between µ and ν is the solution to the following linear program. OT(µ, ν; M):= min DISPLAYFORM1 Here, the optimal T ∈ R n×m is referred to as the transportation matrix: T ij denotes the optimal amount of mass to move from point x i to point y i. Intuitively, OT is concerned with the problem of moving goods from factories to shops in such a way that all the demands are satisfied and the overall transportation cost is minimal. When G = R d and the cost is defined with respect to a metric D G over G (i.e., M ij = D G (x i, y j) p for any i, j), OT defines a distance between empirical probability distributions. This is the p-Wasserstein distance, defined as DISPLAYFORM2 In most cases, we are only concerned with the case where p = 1 or 2.The cost of exactly solving OT problem scales at least in O(n 3 log(n)) (n being the cardinality of the support of the empirical measure) when using network simplex or interior point methods. we consider the entropy regularized Wasserstein distance, W λ p (µ, ν). The above problem can then be solved efficiently using Sinkhorn iterations, albeit at the cost of some approximation error. The regularization strength λ ≥ 0 controls the accuracy of approximation and recovers the true OT for λ = 0. The cost of the Sinkhorn algorithm is only quadratic in n at each iteration. Further on in our discussion, we will make use of the notion of averaging in the Wasserstein space. More precisely the Wasserstein barycenter, introduced by , is a probability measure that minimizes the sum of (p-th power) Wasserstein distances to the given measures. Formally, given N measures {ν 1, . . ., ν N} with corresponding weights η = {η 1, . . ., η N} ∈ Σ N, the Wasserstein barycenter can be written as follows: DISPLAYFORM3 We similarly consider the regularized barycenter B λ p, using entropy regularized Wasserstein distances W λ p in the above minimization problem, following. Employing the method of iterative Bregman projections , we obtain an approximation of the solution at a reasonable computational cost. In this section, we elaborate on both the distributional and the point estimate that we attach to each word, as mentioned in the introduction. A common method in NLP to empirically estimate the probability p(w|c) of occurrence of a word w in some context c, is to compute the number of times the word w co-occurs with context c relative to the total number of times context c appears in the corpus. The context c could be a particular word, phrase, sentence or other definitions of co-occurrence of interest. Distributional Estimate. For a word w, its distributional estimate is built from a histogram over the set of contexts C, and an embedding of these contexts into a space G.A natural way to build this histogram is to maintain a co-occurrence matrix between words in our vocabulary and all possible contexts, such that its each entry indicates how often a word and context occur in an interval (or window) of a fixed size L. Then, the bin values ((H w) c ) c∈C of the histogram (H w) for a word w, can be viewed as the row corresponding to w in this co-occurrence matrix. In Section 5, we discuss how to reduce the number of bins in the histogram, and possible modifications of the co-occurrence matrix to improve associations. The simplest embedding of contexts is into the space of one-hot vectors of all the possible contexts. However, this induces a lot of redundancy in the representation and the distance between contexts does not reflect their semantics. A classical solution would be to instead find a dense lowdimensional embedding of contexts that captures the semantics, possibly using techniques such as SVD or deep neural networks. We denote by V = (v c) c∈C an embedding of the contexts into this low-dimensional space G ⊂ R d, which we refer to as the ground space. (We will consider prototypical cases of how this metric can be obtained in Sections 6 and 7.)Combining the histogram H w and the embedding V, we represent the word w by the following empirical distribution: DISPLAYFORM0 Recall that δ(v c) denotes the Dirac measure at the position v c of the context c. We refer to this representation (Eq.) as the distributional estimate of the word. Together with its distributional estimate, the word w also has an associated point estimate v w when it occurs in the sense of a context, in the form of its position (or embedding) in the ground space. This is what we mean by attaching the distributional and point estimate to each word. Distance. If we equip the ground space G with a meaningful metric D G, then we can subsequently define a distance between the representations of two words w i and w j, as the solution to the following optimal transport problem: DISPLAYFORM1 Intuitively, two words are similar in meaning if the contexts of one word can be easily or cheaply transported to the contexts of the other word, with this ease of transportation being measured by D G. This idea still remains in line with the distributional hypothesis BID11 ) that words in similar contexts have similar meanings, but provides a unique way to quantify it. Interpretation. In fact, both of these estimates are closely tied together and required to serve as an effective representation. For instance, if we only have the distributional estimates, then we may have Here, we pick four contexts at random from a list of top 20 contexts (in terms of PPMI) for the two histograms. Then using the regularized Wasserstein distance (as in Eq. ), we plot the obtained transportation matrix (or commonly called transport map) T as above. Note how'ivory' adjusts its movement towards'skin' (as in skin color) to allow'poaching' to be easily moved to'endangered' as going to other contexts of'mammal' is costly for'poaching', thus capturing a global perspective.two words such as'tennis' and'football' which occur in the contexts of {court, penalty, judge} and {stadium, foul, referee} respectively. While these contexts are mutually disjoint, they are quite close in meaning. Now there could be a third word such as'law' which occurs in the exact same contexts as tennis. So considering the distributional estimate alone, without making use of the point estimates of context, would lead us to have a smaller distance between tennis and law as compared to tennis and football. Whereas, if we only considered the point estimates, then we would lose much of the uncertainty associated about the contexts in which they occur, except for maybe the restricted information of neighboring points in the ground space. This is made clear in a related illustration shown in Figure 2.The family of problems where such a representation can be used is not restricted to entities pertaining to NLP: the framework can be similarly used in any domain where a co-occurrence structure exists between entities and their contexts. For instance, in the case of movie recommendation where users correspond to the entities and movies to the contexts. Lastly, this connection with optimal transport allows us to utilize its rich theoretical and algorithmic toolkit towards important problems in NLP. In the next section, we discuss a concrete framework of how this can be applied and in Section 6 and 7, we detail how the tasks of sentence representation and hypernymy detection can be effectively carried out with this framework. For the sake of brevity, we present the framework for the case where contexts consist of single words. Making associations better. Let's say that a word is considered to be a context word if it appears in a symmetric window of size L around the target word (the word whose distributional estimate we seek). Now, the co-occurrence matrix is between the words of our vocabulary, with rows and columns indicating target words and context words respectively. While each entry of this matrix reflects the co-occurrence count, it may not suggest a strong association between the target and its context. For instance, in the sentence "She prefers her coffee to be brewed fast than being perfect", there is a stronger association between'coffee' and'brewed' rather than between'coffee' and'her', although the co-occurrence counts alone might imply the opposite. Hence, to handle this we consider the well-known Positive Pointwise Mutual Information (PPMI) matrix , whose entries are as follows: DISPLAYFORM0 The PPMI entries are non-zero when the joint probability of the target and context words cooccurring together is higher than the probability when they are independent. Typically, these probabilities are estimated from the co-occurrence counts #(w, c) in the corpus and lead to DISPLAYFORM1 where, #(w) = c #(w, c), #(c) = w #(w, c) and |Z| = w c #(w, c). Also, it is known that PPMI is biased towards infrequent words and assigns them a higher value. A common solution is to smoothen 2 the context probabilities by raising them to an exponent of α lying between 0 and 1. Levy and Goldberg (2014b) have also suggested the use of the shifted PPMI (SPPMI) matrix where the shift 3 by log(s) acts like a prior on the probability of co-occurrence of target and context pairs. These variants of PPMI enable us to extract better semantic associations from the cooccurrence matrix. Finally, we define DISPLAYFORM2 Hence, the bin values for our histogram in Eq. are formed as: DISPLAYFORM3 Computational considerations. The view of optimal transport between histograms of contexts introduced in Eq. offers a pleasing interpretation (see FIG0 . However, it might still be a computationally intractable in its current formulation. Indeed the number of possible contexts can be as large as the size of vocabulary (if the contexts are just single words) or even exponential (if contexts are considered to be phrases, sentences and otherwise). For instance, even with the use of SPPMI matrix, which also helps to sparsify the co-occurrences, the cardinality of the support the word histograms still varies from 10 3 to 5 × 10 4 context words, when considering a vocabulary of size around 2 × 10 5. This is a problem because the Sinkhorn algorithm for regularized optimal transport (Section 3), scales roughly quadratically in the histogram size and the ground cost matrix can also become prohibitive to store in memory, for the range of histogram sizes mentioned. One possible fix is to instead consider a few representative contexts in this ground space. The hope is that with the dense low-dimensional embeddings and a meaningful metric between them, we may not require as many contexts as needed before. For instance, this can be achieved by clustering the contexts with respect to metric D G. Besides the computational gain, the clustering will lead us to consider this transport between more abstract contexts. This will although come at the loss of some interpretability. Another alternative for dealing with this computational issue could be to consider stochastic optimal transport techniques BID6, where the intuition would be to randomly sample a subset of contexts while considering this transport. But we leave that direction for a future work. Now, consider that we have obtained K contexts, each representing some part C k of the set of contexts C. The histogram for word w with respect to these contexts can then be written as DISPLAYFORM4 Hereṽ k ∈Ṽ denotes the point estimate of the k th representative context, and (H w) k are the new bin values for the histogram similar to that in Eq., but with respect to these parts, DISPLAYFORM5 Furthermore, in certain cases 4, it can be important to measure the relative portion of C k's SPPMI (Eq. FORMULA12) that has been used towards a word w. Otherwise the process of making the histogram unit sum in Eq. will misrepresent the actual underlying contribution (check Eq. in Appendix A 4 when the SPPMI contributions towards the partitions (or clusters) have a large variance. for more details): DISPLAYFORM6 Summary. While we detailed the case of context as single words, this framework can be extended in a similar manner to take into account other contexts such as bi-grams, tri-grams, n-grams or other abstract semantic concepts. Building this suggested representation comes at almost free cost during the typical learning of point-estimates for an NLP task, as the co-occurrence counts can simply be maintained while going through the corpus. GloVe even constructs the co-occurrence matrix explicitly as a precursor to learning the point-estimates. Traditionally, the goal of this task is to develop a representation for sentences, that captures the semantics conveyed by it. Most unsupervised representations proposed in the past rely on the composition of vector embeddings for the words, through either additive, multiplicative, or other ways (; ;). We propose to represent sentences as probability distributions to better capture the inherent uncertainty and polysemy. Our belief is that the meaning of a sentence can be understood as a concept that best explains the simultaneous occurrence of the words in it. We hypothesize that a sentence, S = (w 1, w 2, . . ., w N), can be efficiently represented via the Wasserstein barycenter (see Eq.) of distributional estimates of the words in the sentence, i.e., DISPLAYFORM0 which is itself again a distribution over G. Yet another interesting property is the nonassociativity 5 of the barycenter operation. This can be utilized to take into account the order of the words in a sentence. For now, we restrict our focus on exploring how well barycenters of words taken all at once can represent sentences and this direction is left for future work. Interestingly, the classical weighted averaging of point-estimates (BID3 BID1 BID0 BID2 . The objective here is to predict how similar or dissimilar are two sentences in their meanings. Since with barycenter representation as in Eq. FORMULA13, each sentence is also a histogram of contexts, we can again make use of optimal transport to define the distance between two sentences S 1 and S 2, DISPLAYFORM1 As a ground metric, we consider the Euclidean distance between the point estimates of words. This point estimate for a word is its embedding in the context space and can be obtained with the help of Word2vec or GloVe . For, this task we train the word embeddings on the Toronto Book Corpus via GloVe and in the process also gain the distributional estimates of words for free. Since the word embeddings in these methods are constructed so that similar meaning words are close in cosine similarity, we find the representative points by performing K-means clustering with respect to this similarity. We benchmark our performance against SIF (Smooth Inverse Frequency) method from Arora et al. FORMULA12 who regard it as a "simple but toughto-beat baseline", as well as against the common Bag of Words (BoW) averaging. For this experiment, we use SIF's publicly available implementation 6 and perform the evaluation using SentEval 6 https://github.com/PrincetonML/SIF . Table 1 shows that we always beat BoW and SIF with weighted averaging on all tasks. Further, we perform better than the best variant of SIF (which in addition removes the 1 st principal component) on 3 out of 5 tasks. Also, on the other two tasks we still perform competitively and achieve an overall gain over their best variant with K = 300 clusters (refer to TAB5 in Appendix A for detailed ). Note that, the hyperparameters for SIF are taken to be the best ones separately for each task. Whereas we used the same set of hyperparameters for all the above tasks and the PPMI specific hyperparameters haven't been tuned much, but this should not give us an edge over them. In our comparison, we do not include methods such as Sent2vec , as they are specifically trained to work well on the given task of sentence representation, and such an approach for training remains outside the scope of current work. Our approach for representing barycenters does not require any additional training and is still able to match and outperform strong baselines for the task of semantic similarity. This highlights the efficacy of proposed representation. In linguistics, hypernymy is a relation between words (or sentences) where the semantics of one word (the hyponym) are contained within that of another word (the hypernym). A simple form of this relation is the is-a relation, e.g., cat is an animal. Hypernymy is a special case of the more general concept of lexical entailment which may be broadly defined as any semantic relations between two lexical items where the meaning of one is implied by the meaning of the other. Detecting lexical entailment relations is relevant for numerous tasks in NLP. Given a database of lexical entailment relations, e.g., containing Roger Federer is a tennis player might help a question answering system an- Table 2: Comparison between entailment vectors and optimal transport / Wasserstein based entailment measure (WE). The scores are AP@all (%). The hyperparameter α refers to the smoothing exponent and s to the shift in the PPMI computation. More datasets are presented in TAB4 Table 3: Comparison between entailment vectors, optimal transport / Wasserstein based entailment measure (WE) and other state-of-the-art methods. GE+C and GE+KL are Gaussian embeddings with cosine similarity and negative KL-divergence. The scores for GE+C, GE+KL, and DIVE + C·∆S are taken from as we use the same evaluation setup. The scores are again AP@all (%).swering the question " Who is Switzerland's most successful tennis player?". First distributional approaches to detect hyponymy were unsupervised and exploited different linguistic properties of hypernymy (; ; ;). While most of these methods are count-based, word embedding based methods (; ; BID13 have become more popular in recent years. Other approaches represent words by Gaussian distributions and use KL-divergence as a measure of entailment (a;). Especially for tasks like hypernymy detection, these methods have proven to be powerful as they not only capture the semantics but also the uncertainty about various concepts in which the word appears. Using the framework presented in Section 4, we define a measure of entailment as the optimal transport cost (see Eq. ) between associated distributions under a suitable ground cost. For this purpose, we rely on a model that was recently proposed by BID13 BID12 which explicitly models what information is known about a word by interpreting each entry of the embedding as the degree to which a certain feature is present. Based on the logical definition of entailment they derive an approximate inference procedure and an operator measuring the degree of entailment between two so-called entailment vectors defined as follows: DISPLAYFORM0, where the sigmoid function σ and log are applied component-wise on the embeddings v y, v x. Thus, our choice for the ground cost D on the basis of this entailment operator is D DISPLAYFORM1 This asymmetric and not necessarily positive ground cost illustrates that our framework can be flexibly used with an arbitrary cost function defined on the ground space. Evaluation. In total, we evaluated our method on 9 standard datasets: BLESS , EVALution , Lenci/Benotto , Weeds , Henderson 7 (, Baroni , Kotlerman , Levy and Turney . As an evaluation metric, we use average precision AP@all . For comparison we also report the performance of the entailment embeddings that were trained as described in BID12 8. Following we pushed any OOV (out-ofvocabulary) words in the test data to the bottom of the list, effectively assuming that the word pairs do not have a hypernym relation. Table 2 compares the performance 9 of entailment embeddings and the optimal transport measure based on the ground cost defined in Eq.. Our method yields significant improvements over the entailment embeddings by BID12 on almost all of the datasets. Only on the Baroni dataset, our method performs worse but nevertheless still achieves similar performance as other state-of-the-art methods. It confirms the findings of and : there is no single hypernymy scoring function that performs best on all datasets. Furthermore, on some datasets (EVALution, LenciBenotto, Weeds, Turney) we even outperform or match state-of-theart performance (cf . Table 3), by simply using our framework together with ground cost as defined in Eq..Notably, our method is not specific to the entailment vectors by BID12. It can be used with any embedding vectors and ground cost measuring the degree of entailment, without requiring any additional training. A more accurate ground cost or embedding vectors might even further improve the performance. Furthermore, our training dataset (Wikipedia with 1.7B tokens) and our vocabulary with only 80'000 words are rather small compared to the datasets used, e.g., by (a). We expect to get even better by using a larger vocabulary on a larger corpus. To sum up, we advocate for associating both a distributional and point estimate as a representation for each entity. We show how this allows us to use optimal transport over the set of contexts associated with these entities, in problems with a co-occurrence structure. Further, the framework Aitor Gonzalez-Agirre. 2012. Semeval-2012 In particular, when β = 1, we recover the equation for histograms as in Section 5, and β = 0 would imply normalization with respect to cluster sizes. We make use of the Python Optimal Transport (POT) 12 for performing the computation of Wasserstein distances and barycenters on CPU. For more efficient GPU implementation, we built custom implementation using PyTorch. We also implement a batched version for barycenter computation, which to the best of our knowledge has not been done in the past. The batched barycenter computation relies on a viewing computations in the form of blockdiagonal matrices. As an example, this batched mode can compute around 200 barycenters in 0.09 seconds, where each barycenter is of 50 histograms (of size 100) and usually gives a speedup of about 10x. For all our computations involving optimal transport, we typically use λ around 0.1 and make use of log or median normalization as common in POT to stabilize the Sinkhorn iterations. Clustering: For clustering, we make use of kmcuda's 13 efficient implementation of K-Means algorithm on GPUs. We plan to make all our code (for all these parts) and our pre-computed histograms (for the mentioned datasets) publicly available on GitHub soon. Detailed of the sentence representation and hypernymy detection experiments are listed on the following pages.

Represent the TLDR for finding the corresponding scientific article

Represent each entity based on its histogram of contexts and then Wasserstein is all you need!

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: In this paper, we propose two methods, namely Trace-norm regression (TNR) and Stable Trace-norm Analysis (StaTNA), to improve performances of recommender systems with side information. Our trace-norm regression approach extracts low-rank latent factors underlying the side information that drives user preference under different context. Furthermore, our novel recommender framework StaTNA not only captures latent low-rank common drivers for user preferences, but also considers idiosyncratic taste for individual users. We compare performances of TNR and StaTNA on the MovieLens datasets against state-of-the-art models, and demonstrate that StaTNA and TNR in general outperforms these methods. The boom of user activity on e-commerce and social networks has continuously fueled the development of recommender systems to most effectively provide suggestions for items that may potentially match user interest. In highlyrated Internet sites such as Amazon.com, YouTube, Netflix, Spotify, LinkedIn, Facebook, Tripadvisor, Last.fm, and IMDb, developing and deploying personalized recommender systems lie at the crux of the services they provide to users and subscribers . For example, Youtube, one of the worlds most popular video sites, has deployed a recommender system that updates regularly to deliver personalized sets of videos to users based on their previous or recent activity on site to help users find videos relevant to their interests, potentially keeping users entertained and engaged BID5.Among the vast advancements in deep learning and matrix completion techniques to build recommender systems (Ricci BID21, one of the most imperative aspect of research in such area is to identify latent (possibly low-rank) commonalities that drive specific types of user behaviour. For example, BID6 proposes a deep neural network based matrix factorization approach that uses explicit rating as well as implicit ratings to map user and items into common low-dimensional space. Yet, such variety of low-rank methodologies do not address the impact of idiosyncratic behaviour among buyers, which may potentially skew the overall learned commonalities across user groups. In this work, we propose two multi-task learning methods to improve performances of recommender systems using contextual side information. We first introduce an approach based on trace-norm regression (TNR) that enables us to extract low-rank latent dimensions underlying the side information that drive user preference according to variations in context, such as item features, user characteristics, time, season, location, etc. This is achieved by introducing a nuclear-norm regularization penalty term in the multi-task regression model, and we highlight that such latent dimensions can be thought of as homogeneous behaviour among particular types of user groups. Furthermore, we propose a novel recommender framework called Stable Trace-norm Analysis (StaTNA) that not only captures latent low-rank common drivers for user preference, but also considers idiosyncratic taste for individual users. This is achieved by, in addition to the low-rank penalty, adding a sparsity regularization term to exploit the sparse nature of heterogeneous behaviour. Finally, we test the performance of StaTNA on the MovieLens datasets against state-of-the-art models, and demonstrate that StaTNA and TNR in general outperforms these methods. We first introduce some notation that will be adopted throughout the rest of our work. We let Ω denote the set of all observed entries, and for any matrix Z, definē Z = P Ω (Z) asZ ij = Z ij if (i, j) ∈ Ω and 0 otherwise. We let Y ∈ R n×p, be the final output of the recommender system, X ∈ R n×d includes all side information, and L, S ∈ R d×p represent the common and idiosyncratic effects of side information on users. Both L and S can be considered as mappings from the side information to the recommender response. To be more specific, we take a movie recommender system for example: n can be the number of movies, p is the number of users, so each entry of Y is the predicted rating of each user for every movie, while X represents the features of each movie where each feature is d dimensional. When a new movie comes in, we apply L or/and S to the movie's feature to predict the rating of each existing user's, and recommend the movie to users with high predicted ratings. Before turning to TNR, we first consider a regularized lowrank solution for large-scale matrix completion problems called Soft-Impute BID14, which sheds light upon the key ideas of trace-norm regression. The SoftImpute problem is formulated as the following: DISPLAYFORM0 where · F denotes the Frobenius norm and · * denotes the nuclear norm. In this formulation, we minimize the reconstruction error subject to a bound on the nuclear norm, which serves as a convex relaxation of rank of a matrix and allows us to exploit the low-rank structure of the matrix L.Based on similar ideas, trace-norm regression extends this idea of incorporating regularization on the rank of a matrix in the context of multi-task learning, as it minimizes square loss while penalizing large ranks of the coefficient matrix: DISPLAYFORM1 Similar to our introduction of Soft-Impute and TNR, we first discuss a non contextual model that incorporates both the low-rank matrix L and sparse matrix S, namely the stable principal component pursuit (SPCP) BID24; BID23: DISPLAYFORM0 where · 1 denotes the sum of absolute values for all entries of a matrix. S 1 and L * models sparsity in S and the low-rank structure in L respectively. To further illustrate some intuition for the choice of such norms, we provide an example in the context of foreground- separation in video processing. L can be considered as the stationary , which is low-rank due to the strong correlation between frames; while S can represent foreground objects, which normally occupy only a fraction of the video and hence can be treated as sparse. BID15 Finally, in light of SPCP, we propose a novel framework called Stable Trace-norm Analysis (StaTNA) by adding contextual side information to consideration: DISPLAYFORM1 Note that StaTNA can be considered as a combination of trace norm regression and SPCP, and some theoretical aspects are discussed in BID0. The matrix L captures latent homogeneity among preferences of users, such as an Academy Award winning film would be preferable to many users. On the other hand, idiosyncratic tastes of users are embodied in S, such as some users particularly like horror movies that involve robots and monsters. Note that S can also be considered as a way to be robust to outliers in user behaviour or preference. In the case where X is the identity matrix, this problem is reduced to SPCP. Also, TNR can be considered as a special case of StaTNA by taking λ S = ∞, which explains why StaTNA will in general be more robust compared to TNR. In this subsection we will only briefly discuss the methodologies used in this paper to solve TNR and StaTNA, due to space limitations. We first highlight the computational feasibility for both models since methods such as proximal gradient descent BID16 or (Fast) Iterative Shrinkage-Thresholding Algorithm (FISTA, BID2) can be used to solve these problems with provable (sub)linear convergence rate BID12. For StaTNA, we directly apply FISTA to estimate L and S, and the procedure is detailed in Algorithm 1 in Appendix A.1. Next, as aforementioned, TNR is a special case for StaTNA, so to solve TNR, we simply take λ S = ∞ in Algorithm 1. We also point out that Algorithm 1 may be computationally expensive when the matrix is large. In Appendix A.2 we will propose several modifications to our method for solving TNR and StaTNA which will enable us to improve computational complexity. In our work, we consider the MovieLens 100K and MovieLens 1M datasets. The summary for both datasets are shown in TAB1 Note that MovieLens 1M movies do not include all MovieLens 100k movies. In both datasets, each user has rated at least 20 movies, and ratings are whole numbers in the scale of 1-5. In addition, each dataset is associated with a side information matrix whose rows are indexed by movies and includes 1 feature column denoting the movie category (19 categories in total), along with 1128 columns which denote relevant scores of a movie to provided tags. We pre-process side information to obtain two types of sideinformation matrices. For the first type, we apply one-hotencoding to the categorical feature and obtain 19 categorical features for each dataset. The final side information matrix is the concatenation of this categorical representation and relevance scores to given tags, which has dimensions n × 1147 where n is the number of movies in each dataset. For the second type, in addition to one-hot-encoding, we apply GloVe word embeddings with K = 300 BID18 to these categories using an average pooling trick, and in a 300-dimensional categorical representation vector for each movie. The final side information matrix has dimensions n × 1428. For simplicity, we use the suffix "-G" to denote models trained on the second type of side information processed using GloVe word embedding, while models without this suffix will denote models trained on the first type of side information. In this work, we perform two experiments: 1. (Matrix completion with side information) We fill in missing values in the rating matrix using the pre-processed side information matrix, which is the traditional matrix completion problem. For each dataset, we randomly select 20% of observed values in the rating matrix as the test set, and train TNR & StaTNA, and TNR-G & StaTNA-G on the two types of side-information matrices respectively. We use state-of-theart models such as SVD, Sparse FC, and GC-MC as our baseline. Here we point out that these baseline models do not incorporate side information, opposed to our TNR and StaTNA models which are trained on side information. Yet, our experimental will demonstrate that our proposed models, via utilizing side information, will significantly improve performance in this matrix completion task. 2. (Regression) We predict the ratings for new movies based on new side information of the movie, which is similar to the traditional regression problem. For each dataset, we train TNR, TNR-G, StaTNA, StaTNA-G on a randomly selected 80% of all movies and apply trained models on the remaining 20%. Furthermore, both experiments involve two hyperparameters, namely λ L and λ S, which are tuned using 10-fold cross validation. We use Lasso, Elastic Nets, Multi-task Lasso and Multi-task Elastic Nets BID17 as our baseline models. Note that standard Lasso and standard Elastic Nets are trained for each user independently to predict the user's rating for a given movie. The matrix formulations of these baseline models are shown in Table 6 We notice that both TNR and StaTNA do not perform as well as state-of-the-art models in MovieLens 100K. One explanation is that our models require more training data compared to baseline models to fully capture latent structures of both L and S in order to generalize well on test data. Finally, we also point out that StaTNA converges faster, and in general performs better than TNR, as shown in Figure 1 within Appendix C.1.Experiment 2: Regression In the second experiment to predict user ratings for movies, as shown in TAB4. MAE and RMSE for test data in Experiment 2 (regression) for baseline models (Lasso and Elastic Nets (denoted as EN)), TNR and StaTNA using MovieLens 100K and MovieLens 1M. The prefix "MT" is the abbreviation for "Multi-task".MovieLens 100K, all StaTNA and TNR models significantly outperform baseline models, while TNR with GloVe embedding yields the best out-of-sample performance. For MovieLens 1M, StaTNA with GloVe embedding in the best out-of-sample performance compared to all other baseline models including TNR, with and without GloVe embedding. The strong performance of StaTNA and StaTNA-G across both experiments, and especially in MovieLens 1M, indicates that our StaTNA framework provides promising performance guarantees for solving both matrix completion and regression tasks. As mentioned in earlier sections, we are interested in analyzing particular underlying commonalities in user preferences. We achieve this by investigating the principal components of our estimate of the low-rank matrix L, each of which we consider as a common type of user preference. Since our estimated L is of rank 6, we conclude that there are 6 major common types of user preferences, whose component scores (i.e. explained variance percentages) are listed in Table 4, where we observe that the first principal component explains 88.94% of the variability in user ratings. Table 5. Top 12 features of highest absolute weights within the first two principal components (PC1 and PC2). Details of other principle components are shown in TAB9 in Appendinx C.2. Our methodology to solve TNR and StaTNA (i.e. Algorithm 1 in Appendix A.1) may be computationally expensive when the matrix is large since it requires calling a Singular Value Decomposition (SVD) oracle in each iteration of the algorithm. Hence we propose two alternative methods, a FW-T algorithm and a nonconvex reformulation of the problem, to avoid using an SVD oracle. These are detailed in Appendix A.2. Furthermore, our current studies use side information from only one side, namely movie information. Our StaTNA framework can be extended to incorporate side information for both movies and users: DISPLAYFORM0 where U and M denotes users and movies respectively. Moreover, our StaTNA framework is also compatible with neural networks by including nuclear norm and sparse penalties to the objective. We believe that similar formulations will provide us with better performance guarantees, but at the cost of model interpretability. In this section, we discuss the methodologies we use to solve TNR and StaTNA. As mentioned earlier, we use (Fast) Iterative Shrinkage-Thresholding Algorithm (FISTA, BID2) to solve these problems. Before we address the detailed applications of these algorithms in our context to solve TNR and StaTNA, we introduce the following optimization oracles. We define the proximal mapping of the 1 norm as DISPLAYFORM1, whose extension to matrices is obtained by applying the scalar operator to each element. Moreover, we define the proximal mapping of the nuclear norm BID4 BID13 DISPLAYFORM2 V, and Y = U DV is the SVD of matrix Y. Now, using these definitions, we detail the algorithm to solve StaTNA in Algorithm 1. Note that one can also initialize L 0 in both Algorithm 1 as DISPLAYFORM3, where † denotes the pseudo-inverse of a matrix. For StaTNA, we directly apply FISTA to estimate L and S, and the procedures are detailed in Algorithm 1. As aforementioned, TNR is a special case for StaTNA, so to solve TNR, we simply take λ S = ∞ in Algorithm 1, which forces all S k andŜ k to 0. DISPLAYFORM4 In Algorithm 1, we call an SVD oracle in each iteration in FISTA to find the proximal mapping of the nuclear norm, which is computationally expensive when the matrix is large, i.e. the number of movie features and users are large. Here we propose two methods to avoid using an SVD oracle. First, inspired by a scalable algorithm FW-T on SPCP BID10 BID15, we propose a similar FW-T algorithm to solve StaTNA by replacing the proximal mapping of the nuclear norm with a Frank-Wolfe update in each iteration. To be more specific, we consider the following reformulation of StaTNA: DISPLAYFORM0 for some U L and U S such that the optimal solution (L, S) is still feasible to the above problem, i.e. L * ≤ U L and S * ≤ U S. For simplicity, we write the above objective function as g(L, S, t L, t S). In each iteration of the FW-T algorithm, we call the following Frank-Wolfe oracle (Algorithm 2) for L and S respectively. We can see that for the Frank-Wolfe update for matrix L only requires to compute the leading singular pairs for a matrix, which can be achieved by computationally cheap power iteration BID11. In addition, we perform an exact line-search by easily solving a DISPLAYFORM1 The full algorithm for the FW-T is detailed in Algorithm 3.Algorithm 3 FW-T for StatNA DISPLAYFORM2 Second, we propose a nonconvex formulation of the problem suggested by BID1 BID22: DISPLAYFORM3 where U ∈ R d×r, V ∈ R p×r. This problem is nonconvex but smooth, and these two problems are equivalent in the sense that there is a ono-on-one correspondence between the global minima of these two problems BID3 BID7. Since this new formulation has multi-affine structure, according to new given by BID9, we can use ADMM to solve this, which is detailed in Algorithm 4.To apply ADMM on this nonconvex reformulation, we further reformulate the problem as the following DISPLAYFORM4 5: DISPLAYFORM5 8: DISPLAYFORM6 10: DISPLAYFORM7 DISPLAYFORM8 Frobineous norm Table 6. Summary for formulations of models. MTR is the abbreviation for multitask regression, while MC is the abbreviations for matrix completion. DISPLAYFORM9 DISPLAYFORM10 C. Additional Stuff C.1. Experiment Figures (Figures 1 and 2) C.2. Other Principal Components (TAB9 . Top 12 features of highest absolute weights within the third to sixth principal components (PC3, PC4, PC5 and PC6)

Methodologies for recommender systems with side information based on trace-norm regularization

Represent the TLDR for finding the corresponding scientific article!