Question: In this paper we propose to perform model ensembling in a multiclass or a multilabel learning setting using Wasserstein (W.) barycenters. Optimal transport metrics, such as the Wasserstein distance, allow incorporating semantic side information such as word embeddings. Using W. barycenters to find the consensus between models allows us to balance confidence and semantics in finding the agreement between the models. We show applications of Wasserstein ensembling in attribute-based classification, multilabel learning and image captioning generation. These show that the W. ensembling is a viable alternative to the basic geometric or arithmetic mean ensembling. Model ensembling consists in combining many models into a stronger, more robust and more accurate model. Ensembling is ubiquitous in machine learning and yields improved accuracies across multiple prediction tasks such as multi-class or multi-label classification. For instance in deep learning, output layers of Deep Neural Networks(DNNs), such as softmaxes or sigmoids, are usually combined using a simple arithmetic or geometric mean. The arithmetic mean rewards confidence of the models while the geometric means seeks the consensus across models. What is missing in the current approaches to models ensembling, is the ability to incorporate side information such as class relationships represented by a graph or via an embedding space. For example a semantic class can be represented with a finite dimensional vector in a pretrained word embedding space such as GloVe BID28. The models' predictions can be seen as defining a distribution in this label space defined by word embeddings: if we denote p i to be the confidence of a model on a bin corresponding to a word having an embedding x i, the distribution on the label space is therefore p = i p i δ xi. In order to find the consensus between many models predictions, we propose to achieve this consensus within this representation in the label space. In contrast to arithmetic and geometric averaging, which are limited to the independent bins' confidence, this has the advantage of carrying the semantics to model averaging via the word embeddings. More generally this semantic information can be encoded via cost a matrix C, where C ij encodes the dissimilarity between semantic classes i and j, and C defines a ground metric on the label space. To achieve this goal, we propose to combine model predictions via Wasserstein (W.) barycenters BID0, which enables us to balance the confidence of the models and the semantic side information in finding a consensus between the models. Wasserstein distances are a naturally good fit for such a task, since they are defined with respect to a ground metric in the label space of the models, which carry such semantic information. Moreover they enable the possiblity of ensembling predictions defined on different label sets, since the Wasserstein distance allows to align and compare those different predictions. Since their introduction in BID0 W. barycenter computations were facilitated by entropic regularization BID6 and iterative algorithms that rely on iterative Bregman projections BID2. Many applications have used W. barycenters in Natural Language Processing (NLP), clustering and graphics. We show in this paper that W. barycenters are effective in model ensembling and in finding a semantic consensus, and can be applied to a wide range of problems in machine learning (Table 1).The paper is organized as follows: In Section 2 we revisit geometric and arithmetic means from a geometric viewpoint, showing that they are 2 and Kullback Leibler divergence KL (extended KL divergence) barycenters respectively. We give a brief overview of optimal transport metric and W. barycenters in Section 3. We highlight the advantages of W. barycenter ensembling in terms of semantic smoothness and diversity in Section 4. Related work on W. barycenters in Machine learning are presented in Section 5. Finally we show applications of Wasserstein ensembling on attribute based classification, multi-label learning and image captioning in Section 6. Normalized and Unnormalized predictions Ensembling. In deep learning, predictions on a label space of fixed size M are usually in one of two forms: a) normalized probabilities: in a multiclass setting, the neural network outputs a probability vector (normalized through softmax), where each bin corresponds to a semantic class; b) unnormalized positive scores: in a multi-label setting, the outputs of M independent logistic units are unnormalized positive scores, where each unit corresponds to the presence or the absence of a semantic class. Model ensembling in those two scenarios has long history in deep learning and more generally in machine learning BID3 BID13 ) as they lead to more robust and accurate models. As discussed in the introduction, two methods have been prominent in model ensembling due to their simplicity: majority vote using the arithmetic mean of predictions, or consensus based using the geometric mean. Revisiting Arithmetic and Geometric Means from a geometric viewpoint. Given m predictions µ, and weights λ ≥ 0 such that m =1 λ = 1, the weighted arithmetic mean is given byμ a = m =1 λ µ, and the weighted geometric mean byμ g = Π m =1 (µ λ).It is instructive to reinterpret the arithmetic and geometric mean as weighted Frechet means (Definition 1) . 2 2 (the 2 Euclidean distance). A less known fact is that the geometric mean corresponds to a Frechet Mean for d = KL, where KL is the extended KL divergence to unnormalized measures: KL(p, q) = i p i log pi qi − p i + q i. We give proofs and properties of arithmetic and geometric mean in Appendix F.Following this geometric viewpoint, in order to incorporate the semantics of the target space in model ensembling, we need to use a distance d that takes advantage of the underlying geometry of the label space via a cost matrix C when comparing positive measures. Optimal transport (OT) metrics such as Wasserstein-2 have this property since they are built on an explicit cost matrix defining pairwise distance between the semantic classes. In this paper we propose to use the Frechet means with Wasserstein distance (d = W 2 2) for model ensembling, i.e. use Wasserstein barycenters BID0 for model ensembling: DISPLAYFORM0 Intuitively, the barycenter looks for a distribution ρ (a histogram) that is close to all the base distributions µ in the Wasserstein sense. In our context transporting the consensus ρ to each individual model µ should have a minimal cost, where the cost is defined by the distance in the word embedding space. Wasserstein distances were originally defined between normalized probability vectors (Balanced OT) BID36 ), but they have been extended to deal with unnormalized measures and this problem is referred to as unbalanced OT BID5 BID14. Motivated by the multi-class and the multi-label ensembling applications, in the following we present a brief overview of W. barycenters in the balanced and unbalanced cases., p represents histograms on source label space Ω S = {x i ∈ R d, i = 1 . . . N}, for e.g words embeddings. Consider similarly q ∈ ∆ M representing histograms whose bins are defined on a target label space Ω T = {y j ∈ R d, j = 1 . . . M}. Consider a cost function c(x, y), (for example c(x, y) = x − y 2 ). Let C be the matrix in ∈ R N ×M such that C ij = c(x i, y j). 1 N denotes a vector with all ones. Let γ ∈ R N ×M be a coupling matrix whose marginals are p and q such that: DISPLAYFORM0 The optimal transport metric is defined as follows: DISPLAYFORM1 When c(x, y) = x − y 2 2, this distance corresponds to the so called Wasserstein−2 distance W 2 2.Unbalanced OT. When p and q are unnormalized and have different total masses, optimal transport metrics have been extended to deal with this unbalanced case. The main idea is in relaxing the set Π(p, q) using a divergence such as the extended KL divergence: KL. BID5 define for λ > 0 the following generalized Wasserstein distance between unnormalized measures: DISPLAYFORM2 Throughout the paper we consider m discrete prediction vectors µ ∈ R N +, = 1... m defined on a discrete space (word embeddings for instance) DISPLAYFORM0 We refer to Ω S as source spaces. Our goal is to find a consensus predictionμ DISPLAYFORM1 Balanced W. Barycenters: Normalized predictions. The W. barycenter BID0 of normalized predictions is defined as follows:μ w = arg min ρ m =1 λ W (ρ, µ), for the Wasserstein distance W defined in equation. Hence one needs to solve the following problem, for m coupling matrices γ, = 1... m: DISPLAYFORM2 Unbalanced W. Barycenters: Unnormalized predictions. Similarly the W. barycenter of unnormalized predictions is defined as follows:μ w = arg min ρ m =1 λ W unb (ρ, µ), for the Generalized Wasserstein distance W unb defined in equation. Hence the unbalanced W. barycenter problem BID5 amounts to solving, for m coupling matrices γ, = 1... m: DISPLAYFORM3 3.3 COMPUTATION VIA ENTROPIC REGULARIZATION AND PRACTICAL ADVANTAGES Entropic Regularized Wasserstein Barycenters Algorithms. The computation of the Wasserstein distance grows super-cubicly in the number of points. This issue was alleviated by the introduction of the entropic regularization BID6 to the optimization problem making it strongly convex. Its solution can be found using scaling algorithms such as the so called Sinkhorn algorithm. For any positive matrix γ, the entropy is defined as follows: DISPLAYFORM4 The entropic regularized OT distances in the balanced and unbalanced case become, for a hyperparameter ε > 0: DISPLAYFORM5 for ε → 0, W ε and W unb,ε converge to the original OT distance, and for higher value of ε we obtain the so called Sinkhorn divergence that allows for more diffuse transport between p and q. Balanced and unbalanced W. barycenters can be naturally defined with the entropic regularized OT distance as follows: DISPLAYFORM6 respectively. This regularization leads to simple iterative algorithms BID2 BID5 ) (for more details we refer the interested reader to BID5 and references therein) for computing W. barycenters that are given in Algorithms 1 and 2.Algorithm 1: Balanced Barycenter for Multiclass Ensembling BID2 Inputs: DISPLAYFORM7 Algorithm 2: Unbalanced Barycenter for Multilabel Ensembling BID5 ) DISPLAYFORM8 We see that the output of Algorithm 1 is the geometric mean of K u, = 1... m, where K is a Gaussian kernel with bandwidth ε the entropic regularization parameter. Note v *, = 1... m the values of v at convergence of Algorithm 1. The entropic regularized W. barycenter can be written as follows: exp DISPLAYFORM9. We see from this that K appears as matrix product multiplying individual models probability µ and the quantities v * related to Lagrange multipliers. This matrix vector product with K ensures probability mass transfer between semantically related classes i.e between items that has entries K,ij with high values. Remark 1 (The case K = K = I). As the kernel K in Algorithm 1 approaches I (identity) (this happens when ε → 0), the alternating Bregman projection of BID2 for balanced W. barycenter converges to the geometric meanμ g = Π m =1 (µ) λ.We prove this in Appendix D. When K = I the fixed point of Algorithm 1 reduces to geometric mean, and hence diverges from the W. barycenter. Note that K approaches identity as ε → 0, and in this case we don't exploit any semantics. Wasserstein Ensembling in Practice. Table 1 gives a summary of machine learning tasks that can benefit from Wasserstein Ensembling, and highlights the source and target domains as well as the corresponding kernel matrix K. In the simplest case Ω S = Ω T and N = M for all, this corresponds to the case we discussed in multi-class and multi-labels ensemble learning, W. barycenters allows to balance semantics and confidence in finding the consensus. The case where source and target spaces are different is also of interest, we give here an application example in attribute based classification: µ corresponds to prediction on a set of attributes and we wish to make predictions through the W. barycenter on a set of labels defined with those attributes. See Section 6.1. as we use beam search on the predictions, diversity and smoothness of the predictions become key to the creativity and the composition of the sequence generator in order to go beyond "baby talk" and vanilla language based on high count words in the training set. Hence we need to increase the entropy of the prediction by finding a semantic consensus whose predictions are diverse and smooth on semantically coherent concepts without compromising accuracy. We will show in the following proposition that the W. barycenter allows such aggregation: Proposition 1 (Properties of Wasserstein Barycenters). Let ν be the target distribution (an oracle) defined on a discrete space Ω = {x 1, . . . x K, x j ∈ R d} (word embedding space) and µ, = 1... m be m estimates of ν. Assume W 2 2 (µ, ν) ≤ ε. The W. barycenterμ w of {µ} satisfies the following: 1) Semantic Accuracy (Distance to an oracle). We have: DISPLAYFORM10 2) Diversity. The diversity of the W. barycenter depends on the diversity of the models with respect to the Wasserstein distance (pairwise Wasserstein distance between models): DISPLAYFORM11 3) Smoothness in the embedding space. Define the smoothness energy E (ρ) = DISPLAYFORM12 The W. barycenter is smoother in the embedding space than the individual models. DISPLAYFORM13 Proof. The proof is given in Appendix F.We see from Proposition 1 that the W. barycenter preserves accuracy, but has a higher entropy than the individual models. This entropy increase is due to an improved smoothness on the embedding space: words that have similar semantics will have similar probability mass assigned in the barycenter. The diversity of the barycenter depends on the Wasserstein pairwise distance between the models: the W. barycenter output will be less diverse if the models have similar semantics as measured by the Wasserstein distance. The proof of proposition 1 relies on the notion of convexity along generalized geodesics of the Wasserstein 2 distance BID0. Propositions 2 and 3 in Appendix F give similar for geometric and arithmetic mean, note that the main difference is that the guarantees are given in terms of KL and 2 respectively, instead of W 2.In order to illustrate the diversity and smoothness of the W. barycenter, we give here a few examples of the W. barycenter on a vocabulary of size 10000 words, where the cost matrix is constructed from word synonyms ratings, defined using Power Thesaurus or using GloVe word embeddings BID28. We compute the W. barycenter (using Algorithm 1) between softmax outputs of 4 image captioners trained with different random seeds and objective functions. Figure 4 shows the W. barycenter as well as the arithmetic and geometric mean. It can be seen that the W. barycenter has higher entropy and is smooth along semantics (synonyms or semantics in the GloVe space) and hence more diverse than individual models. Table 2 shows top 15 words of barycenter, arithmetic and geometric means, from which we see that indeed the W. barycenter outputs clusters according to semantics. In order to map back the words x j that have high probability in the W. barycenter to an individual model, we can use the couplings γ as follows: γ ij is the coupling between word j in the barycenter and word i in model. Examples are given in supplement in Table 2: Sample output (top 15 words) of W. barycenter (Algorithm 1), arithmetic and geometric means based on four captioner models. Each row shows a word and a corresponding probability over the vocabulary (as a percentage). W. Barycenter has higher entropy, spreading the probability mass over the synonyms and words related to the top word "car" and downweights the irrelevant objects (exploiting the side information K). Simple averaging techniques, which use only the confidence information, mimick the original model outputs. Figure 4 in Appendix gives a histogram view. Controllable Entropy via Regularization. As the entropic regularization parameter ε increases the distance of the kernel K from identity I increases and the entropy of the optimal couplings γ, (H(γ)) increases as well. Hence the entropy of entropic regularized W. Barycenter is controllable via the entropic regularization parameter ε. In fact since the barycenter can be written asμ w = γ 1 N, one can show that (Lemma 2 in Appendix): DISPLAYFORM14 As epsilon increases the right-hand side of the inequality increases and so does H(μ w). This is illustrated in Tables 3 and 8, we see that the entropy of the (entropic regularized) W. barycenter increases as the distance of the kernel K to identity increases (K − I F increases as ε increases) and the output of the W. barycenter remains smooth within semantically coherent clusters. Table 3: Controllable Entropy of regularized Wasserstein Barycenter (Algorithm 1). Output (top 15 words) for a synonyms-based similarity matrix K under different regularization ε (which controls the distance of K to identity I, K − I F). As ε decreases, K − I F also decreases, i.e., K approaches identity matrix, and the entropy of the output of Algorithm 1 decreases. Note that the last column, corresponding to very small entropic regularization, coincides with the output from geometric mean in FIG3 (for K = I, the Algorithm 1 outputs geometric mean as a barycenter). Wasserstein Barycenters in Machine Learning. Optimal transport is a relatively new comer to the machine learning community. The entropic regularization introduced in BID6 fostered many applications and computational developments. Learning with a Wasserstein loss in a multilabel setting was introduced in BID14, representation learning via the Wasserstein discriminant analysis followed in BID12. More recently a new angle on generative adversarial networks learning with the Wasserstein distance was introduced in (; BID15 BID31 . Applications in NLP were pioneered by the work on Word Mover Distance (WMD) on word embeddings of BID23. Thanks to new algorithmic developments BID7 BID2 W. barycenters have been applied to various problems: in graphics BID35, in clustering , in dictionary learning BID33, in topic modeling , in bayesian averaging BID30, and in learning word and sentences embeddings BID26 BID27 etc. Most of these applications of W. barycenter focus on learning balanced barycenters in the embedding space (like learning the means of the clusters in clustering), in our ensembling application we assume the embeddings given to us (such as GloVe word embedding) and compute the barycenter at the predictions level. Finally incorporating side information such as knowledge graphs or word embeddings in classification is not new and has been exploited in diverse ways at the level of individual model training via graph neural networks BID25 BID9, in the framework of W. barycenter we use this side information at the ensemble level. In this Section we evaluate W. barycenter ensembling in the problems of attribute-based classification, multi-label prediction and in natural language generation in image captioning. As a first simple problem we study object classification based on attribute predictions. We use Animals with Attributes which has 85 attributes and 50 classes. We have in our experiments 2 attributes classifiers to predict the absence/presence of each of the 85 attributes independently, based on resnet18 and resnet34 BID18 input features while training only the linear output layer (following the details in Section 6.2). We split the data randomly in 30322 / 3500 / 3500 images for train / validation / test respectively. We train the attribute classifiers on the train split. Based on those two attributes detectors we would like to predict the 50 categories using unbalanced W. barycenters using Algorithm 2. Note that in this case the source domain is the set of the 85 attributes and the target domain is the set of 50 animal categories. For Algorithm 2 we use a columnnormalized version of the binary animal/attribute matrix as K matrix (85 × 50), such that per animal the attribute indicators sum to 1. We selected the hyperparameters ε = 0.3 and λ = 2 on the validation split and report here the accuracies on the test split. Table 4: Attribute-based classification. The W. barycenter ensembling achieves better accuracy by exploiting the cross-domain similarity matrix K, compared to a simple linear-transform of probability mass from one domain to another as for the original models or their simple averages. As a baseline for comparison, we use arithmetic mean (μ a) and geometric mean (μ g) ensembling of the two attribute classifiers resnet18 and resnet34. Then, using the same matrix K as above, we define the probability of category c (animal) as p(c|µ) = K μ (forμ =μ a andμ g resp.). We see from Table 4 that W. barycenter outperforms arithmetic and geometric mean on this task and shows its potential in attribute based classification. For investigating W. barycenters on a multi-label prediction task, we use MS-COCO BID24 with 80 objects categories. MS-COCO is split into training (≈82K images), test (≈35K), and validation (5K) sets, following the Karpathy splits used in the community BID20. From the training data, we build a set of 8 models using'resnet18' and'resnet50' architectures BID18. To ensure some diversity, we start from pretrained models from either ImageNet BID8 ) or Places365 (. Each model has its last fully-connected ('fc') linear layer replaced by a linear layer allowing for 80 output categories. All these pretrained models are fine-tuned with some variations: The'fc' layer is trained for all models, some also fine-tune the rest of the model, while some fine-tune only the'layer4' of the ResNet architecture. These variations are summarized in Table 5. Training of the'fc' layer uses a 10 −3 learning rate, while all fine-tunings use 10 −6 learning rate. All multi-label trainings use ADAM BID21 with (β 1 = 0.9, β 2 = 0.999) for learning rate management and are stopped at 40 epochs. Only the center crop of 224 * 224 of an input image is used once its largest dimension is resized to 256. Table 5: Description of our 8 models built on MS-COCO Evaluation Metric. We use the mean Average Precision (mAP) which gives the area under the curve of P = f (R) for precision P and recall R, averaged over each class. mAP performs a sweep of the threshold used for detecting a positive class and captures a broad view of a multi-label predictor performance. Performances for our 8 models are reported in TAB5. Precision, Recall and F1 for micro/macro are given in TAB10. Our individual models have reasonable performances overall. Arithmetic and geometric means offer direct mAP improvements over our 8 individual models. For unbalanced W. barycenter, the transport of probability mass is completely defined by its matrix K = K in Algorithm 2. We investigated multiple K matrix candidates by defining K(i, j) as (i) the pairwise GloVe distance between categories, (ii) pairwise visual word2vec embeddings distance, (iii) pairwise co-occurence counts from training data. In our experience, it is challenging to find a generic K that works well overall. Indeed, W. barycenter will move mass exactly as directed by K. A generic K from prior knowledge may assign mass to a category that may not be present in some images at test time, and get harshly penalized by our metrics. A successful approach is to build a diagonal K for each test sample based on the top-N scoring categories from each model and assign the average of model posteriors scores K(i, i) = 1 M m p m (i|x) for image x and category i. If a category is not top scoring, a low K(i, i) = ζ value is assigned to it, diminishing its contribution. It gives W. barycenter the ability to suppress categories not deemed likely to be present, and reinforce the contributions of categories likely to be. This simple diagonal K gives our best when using the top-2 scoring categories per model (the median number of active class in our training data is about 2) and outperforms arithmetic and geometric means as seen in TAB5. In all our experiments, W. barycenters parameters {ε, λ} in Algorithm 2 and ζ defined above were tuned on validation set (5K). We report on MS-COCO test set (≈35K). In this task of improving our 8 models, W. barycenter offers a solid alternative to commonly used arithmetic and geometric means. Appendix B.2 shows that non-uniform weighting further improves W. ensembling performance. In this task the objective is to find a semantic consensus by ensembling 5 image captioner models. The base model is an LSTM-based architecture augmented with the attention mechanism over the image. In this evaluation we selected captioners trained with cross entropy objective as well as GAN-trained models BID10. The training was done on COCO dataset BID24 using data splits from BID19: training set of 113k images with 5 captions each, 5k validation set, and 5k test set. The size of the vocabulary size is 10096 after pruning words with counts less than 5. The matrix K = K in Algorithm 1 was constructed using word similarities, defined based on (i) GloVe word embeddings, so that K = exp(−C/ε), where cost matrix C is constructed based on euclidean distance between normalized embedding vectors; and (ii) synonym relationships, where we created K based on the word synonyms graph and user votes from Power Thesaurus. The model prediction µ, for = 1,..., 5 was selected as the softmax output of the captioner's LSTM at the current time step, and each model's input was weighted equally: λ = 1/m. Once the barycenter p was computed, the was fed into a beam search (beam size B = 5), whose output, in turn, was then given to the captioner's LSTM and the process continued until a stop symbol (EOS) was generated. In order to exploit the controllable entropy of W. barycenter via the entropic regualrization parameter ε, we also decode using randomized Beam search of BID34, where instead of maintaining the top k values, we sample D candidates in each beam. The smoothness of the barycenter in semantic clusters and its controllable entropy promotes diversity in the ing captions. We baseline the W. barycenter ensembling with arithmetic and geometric means.. The x-axis shows K − I F, which corresponds to a different regularization parameter ε (varied form 1 to 50). We can see that for topK beam search (left panel) the further K is from the identity matrix, the larger the similarity neighborhood of each word, the more diverse are the generated captions (the barycenter has higher entropy), while still remaining semantically close to the ground truth. On the other hand, for randomized beam search (right panel), it is important to maintain a smaller similarity neighborhood, so that the generated sentences are not too different from the referenced ground truth. Controllable entropy and diversity. FIG0 show the comparison of the ensembling methods on the validation set using topK and randomized beam search. The x-axis shows K −I F, which corresponds to a different regularization ε (varied form 1 to 50). We report two n-gram based metrics: CIDEr and SPICE scores, as well as the WMD (Word Mover Distance) similarity BID23, which computes the earth mover distance (the Wasserstein distance) between the generated and the ground truth captions using the GloVe word embedding vectors. In topK beam search, as ε increases, causing the entropy to go up, the exact n-grams matching metrics, i.e., CIDEr and SPICE, deteriorate while WMD remains stable. This indicates that while the barycenter-based generated sentences do not match exactly the ground truth, they still remain semantically close to it (by paraphrasing), as indicated by the stability of WMD similarity. The of the GloVe-based barycenter on the test split of COCO dataset are shown in Table 7. In randomized beam search, the increase in entropy of the barycenter leads to a similar effect of paraphrasing but this works only up to a smaller value of ε, beyond which we observe a significant deterioration of the . At that point all the words become neighbors and in a very diffused barycenter, close to a uniform distribution. This diffusion effect is smaller for the synonyms-based K since there are only a certain number of synonyms for each word, thus the maximum neighborhood is limited. Table 7: Performance of GloVe-based W. barycenter on COCO test split using topK beam search versus Geometric and Arithmetic ensembling. While the generated sentences based on W. barycenter do not match exactly the ground truth (lower CIDEr), they remain semantically close to it, while being more diverse (e.g., paraphrased) as indicated by the higher entropy and stable WMD.Robustness of W. Barycenter to Semantic Perturbations. Finally, the right panel of FIG3, shows the robustness of the W. barycenter to random shuffling of the µ values, within semantically coherent clusters. Note that the size of those clusters increases as K moves away from identity. The show that barycenter is able to recover from those perturbations, employing the side information from K, while both the arithmetic and geometric means (devoid of such information) are confused by this shuffling, displaying a significant drop in the evaluation metrics. Comparison of ensembling methods when the predictions of the input models are shuffled according to the neighborhood structure defined by K. It can be seen that the W. Barycenter ensembling is able to recover from the word shuffling and produce better captions then the simple averaging methods, which are not able to exploit the provided side information. Human Evaluation. We performed human evaluation on Amazon MTurk on a challenging set of images out of context of MS-COCO BID10. We compared three ensembling techniques: arithmetic, geometric and W. barycenter. For W. barycenter we used the similarity matrix K defined by visual word2vec BID22. For the three models we use randomized beam search. We asked MTurkers to give a score for each caption on a scale 1-5 and choose the best captions based on correctness and detailedness. Captions examples are given in Fig. 6 (Appendix). FIG4 shows that W. barycenter has an advantage over the basic competing ensembling techniques. We showed in this paper that W. barycenters are effective in model ensembling in machine learning. In the unbalanced case we showed their effectiveness in attribute based classification, as well as in improving the accuracy of multi-label classification. In the balanced case, we showed that they promote diversity and improve natural language generation by incorporating the knowledge of synonyms or word embeddings. Table 8: Sample output (top 20 words) of barycenter for different similarity matrices K based on GloVe (columns titles denote the distance of K from identity K − I F and corresponding .). Each column shows a word and its corresponding probability over the vocabulary. Note that the last column coincides with the output from geometric mean. Table 8 shows the effect of entropic regularization ε on the ing distribution of the words of W. barycenter using GloVe embedding matrix. As K moves closer to the identity matrix, the entropy of barycenter decreases, leading to outputs that are close/identical to the geometric mean. On the other hand, with a large entropic regularization, matrix K moves away from identity, becoming an uninformative matrix of all 1's. This eventually leads to a uniform distribution which spreads the probability mass equally across all the words. This can be also visualized with a histogram in Figure 5, where the histograms on the bottom represent distributions that are close to uniform, which can be considered as failure cases of W. barycenter, since the image captioner in this case can only generate meaningless, gibberish captions. In TAB1 we show a mapping from a few top words in the barycenter output (for similarity matrix K based on synonyms) to the input models. In other words, each column defines the words in the input models which have the greatest influence on each of top 3 words in the barycenter output. In Figure 6 we present a few captioning examples showing qualitative difference between the considered ensembling techniques. Figure 4: Visualization of the word distributions of W. barycenter, arithmetic and geometric means based on four captioning models, whose input image is shown on top (one of the ground-truth human-annotated captions for this image reads: A police car next to a pickup truck at an intersection). The captioner generates a sentence as a sequence of words, where at each step the output is a distribution over the whole vocabulary. The top four histograms show a distribution over the vocabulary from each of the model at time t = 3 during the sentence generation process. The bottom three histograms show the ing distribution over the vocabulary for the ensembles based on W. Barycenter, arithmetic and geometric means. It can be seen that the W. Barycenter produces high entropy distribution, spreading the probability mass over the synonyms of the word "car" (which is the top word in all the four models), based on the synonyms similarity matrix K.Figure 5: Visualization of the word distributions of W. barycenter for different similarity matrices K based on GloVe (rows denote the distance of K from identity K − I F and corresponding). Large entropic regularization generates K close to uninformative matrices of all 1's. This eventually leads to a barycenter which is close to a uniform distribution spreading the probability mass almost equally across all the words. TAB1: Mapping from a few top words in the barycenter output (for similarity matrix K based on synonyms) to the input models. For each word in the left columns, the remaining columns show the contributing words and the percent of contribution. BA: a television is placed on the curb of the road AM: a TV sits on the side of a street GM: a television sitting on the side of a street GT: an empty sidewalk with an abandoned television sitting alone BA: a car that is parked at the station AM: a car that has been shown in a subway GM: a car that is sitting on the side of a road GT: a car at the bottom of the stair well BA: a person is sitting on the sidewalk with a tent AM: a couple of people sitting on benches next to a building GM: a couple of people sitting on the side of a street GT: a woman is sitting with a guitar near a man that is sitting on the ground in front of a tent BA: a sheep sitting in a car looking out the window AM: a white sheep is sitting in a vehicle GM: a close up of a sheep in a car GT: a sheep sitting at the steering wheel of a car with its hooves on the wheels Figure 6: Examples of captions for several images. BA: Wasserstein Barycenter, AM: Arithmetic mean, GM: Geometric mean, GT: Ground truth. We evaluate our models using micro and macro versions of precision, recall, and F1-measure as covered in multi-label prediction metrics study from . For these measures, a threshold of 0.5 is commonly used to predict a label as positive in the community's published . Macro precision is an average of per-class precisions while micro precision is computed by computing the ratio of all true positives across all image samples over the number of all positive classes in a dataset. Therefore a macro (or per-class) precision'P-C' is defined as 1 C i P i while a micro (or overall precision)'P-O' is defined as i T Pi i T Pi+F Pi where T P i and F P i are true and false positives respectively. Per-class and overall versions for R and F1 are defined similarly. We also employ mean Average Precision (mAP) which gives the area under the curve of P = f (R) averaged over each class. Unlike P,R and F1, mAP inherently performs a sweep of the threshold used for detecting a positive class and captures a broader view of a multi-label predictor's performance. Performances for our 8 models and previously published are reported in TAB5 in the paper. Our models have reasonable performances overall. Ensembling given in Tab. 6 are using uniformly weighted models, i.e. λ = 1 m where m is the number of models. However, in practice, arithmetic and geometric mean ensembling usually use weighted ensembles of models The weights are then optimized and established on a small validation set before being used for ensembling on a test set. A well-known embodiment of this type of approach is Adaboost BID13 where weights are dynamically defined at each pass of training wrt to the accuracy of base models. Here, we follow a much simpler but similar approach by defining the performance of each model as the mean average precision (mAP) on the validation set. mAP is used to define λ such that λ = mAP mAP. λ are then applied to the models' scores for arithmetic, geometric mean and W.Barycenter ensemblings. Tab. 11 reports mAP for each ensembling technique over the MS-COCO test set (35150 images). Note that the λ weights definition is based on the final metric evaluation, mAP in this case. For other tasks such as classification, accuracy or any other valid metric can be employed to compute the λ weights. It must be noted that the weights are computed with respect to the ultimate performance metric at hand. Tab. 11 reveals clearly that such approach of weighting models by their performance benefits arithmetic and W.Barycenter ensembling for this task. Both methods leverage the confidence of the underlying models and the mAP weighting of models will reinforce the contributions of better performing models. Geometric means ensembling is not significantly impacted by non-uniform λ since it is mostly relying on consensus of the models, not their confidence. We conclude that weighting indeed helps performance and keeps a performance advantage for W. Barycenter over the alternatives arithmetic and geometric means. Table 11: multi-label models ensembling mAP on MS-COCO test set (35150 images). Performancebased weighting helps both arithmetic and W.Barycenter ensembling, the latter retaining its performance vantage. At iteration 0, the holds since we have: DISPLAYFORM0 Assume the holds at time t. Let us prove it for t + 1, following the updates of Algorithm 1: DISPLAYFORM1 QED.For ε > 0, Feydy et al BID11 showed recently that the Sinkhorn divergence defines an interpolation between the MMD distance (Maximum mean discrepancy BID16) and the Wasserstein Distance. Hence for ε > 0 Algorithm 1 provides still an interesting solution that can be seen as an interpolation between the original (unregularized) Wasserstein Barycenter and the MMD Barycenter (Frechet barycenter for d = MMD). DISPLAYFORM2 As λ goes to infinity this unbalanced cost converges to the Hellinger distance: We make the following theoretical and practical remarks on how to improve this computational complexity to reach an almost linear dependency on N using low rank approximation of the kernel matrix K, and parallelization on m machines: DISPLAYFORM3 1. Dependency on Maxiter: For the number of iterations we found that Maxiter = 5 is enough for convergence, which makes most of the computational complexity dependent on m and N.2. Dependency on N and low rank approximation: The main computational complexity comes from the matrix vector multiply K u that is of O(N 2). Note that this complexity can be further reduced since the kernel matrix K is often low rank. Therefore we can be written K = ΦΦ where Φ ∈ R N ×k, where k N, which allows to compute this product as follows ΦΦ u that has a lower complexity O(N k). Φ can be computed using Nystrom approximation or random Fourier features. Hence potentially on can get an algorithm with complexity O(mN k), where k has a logarithmic dependency on N. This was studied recently in BID1.3. Dependency on m and parallelization: Regarding the dependency on m, as noted in BID5 BID2 the algorithm is fully parallelizable which would lead to a computational complexity of O(N k) by using simply m machines.4. GPU and Batch version: Practically, the algorithm implemented takes advantage of matrix vector products' speed on GPU. The algorithm can be further accelerated by computing Sinkhorn divergences in batches as pointed in BID11 ). We evaluated the time complexity of the GPU implementation of Wasserstein Barycenters in pytorch on our multi-label prediction experiments using MS-COCO test set (35150 samples). Note that we used a vanilla implementation of Algorithm 2, i.e without parallelization, batching, or low rank approximation. Results and comments for these wall clock timings can be found in Tab. 13. As it can be observed, we need to use Maxiter = 5 on a GPU-V100 to reach below 4ms/image for Wasserstein ensembling. This is not a major overhead and can be further improved as discussed previously by using parallelization, batching and low rank approximation. In TAB14, each timing was done over the whole test set; each timing repeated 5 times. We report means and standard deviations of total wall clock times for ensembling 8 models. Last column on the right is the average timing per image (in ms) for W.Barycenter. The number of W.Barycenter iterations (Maxiter) was varied from 1, 5 and 10 to show its impact. We report timing numbers over two GPU architectures, NVIDIA Tesla K80 and V100. W.Barycenters leverage the GPU while Arithmetic and Geometric means do not. Timings are of the computations of the means themselves, no data fetching or data preparation is included in these timings. As expected, the wall clock time cost for W.Barycenters is several order of magnitude higher than for Arithmetic and Geometric means. The difference of GPU does not impact the Arithmetic and Geometric means as they do not use it in our implementation. The Barycenter computation see a speed up from K80 to V100 as V100 is much better at reducing wall time for longer number of iterations. Proposition 2 (propreties of Geometric Mean). The following properties hold for geometric mean:1. Geometric mean is the Frechet mean of KL. The geometric mean is the Frechet mean with respect to the extended KL divergence: First order optimality condition: DISPLAYFORM0 DISPLAYFORM1 This gives us the : DISPLAYFORM2 Published as a conference paper at ICLR 2019Proof. Let γ ∈ R N ×M + be a coupling between p ∈ ∆ M and q ∈ ∆ N, q j > 0 we have: γ 1 N = p and γ1 M = q, we have: DISPLAYFORM3 DISPLAYFORM4 Now the entropy of the convex combination is higher than convex combination of entropies (the entropy is concave): DISPLAYFORM5 γ ij log(γ ij) − log(q i)q i.Hence: DISPLAYFORM6 γ ij log(γ ij) − H(q)Hence: DISPLAYFORM7 γ ij log(γ ij)

Represent the TLDR for finding the corresponding scientific article.

we propose to use Wasserstein barycenters for semantic model ensembling

Represent the short summary for finding the corresponding scientific article (long)

An empirical study of variational inference based on chi-square divergence minimization, showing that minimizing the CUBO is trickier than maximizing the ELBO


Question: Represent the article for finding a single-sentence TLDR

Question: While deep learning has been incredibly successful in modeling tasks with large, carefully curated labeled datasets, its application to problems with limited labeled data remains a challenge. The aim of the present work is to improve the label efficiency of large neural networks operating on audio data through a combination of multitask learning and self-supervised learning on unlabeled data. We trained an end-to-end audio feature extractor based on WaveNet that feeds into simple, yet versatile task-specific neural networks. We describe several easily implemented self-supervised learning tasks that can operate on any large, unlabeled audio corpus. We demonstrate that, in scenarios with limited labeled training data, one can significantly improve the performance of three different supervised classification tasks individually by up to 6% through simultaneous training with these additional self-supervised tasks. We also show that incorporating data augmentation into our multitask setting leads to even further gains in performance. Deep neural networks (DNNs) are the bedrock of state-of-the-art approaches to modeling and classifying auditory data BID0; van den BID20; ). However, these data-hungry neural architectures are not always matched to the available training resources, and the creation of large-scale corpora of audio training data is costly and time-consuming. This problem is exacerbated when training directly on the acoustic waveform, where input is highdimensional and noisy. While labeled datasets are quite scarce, we have access to virtually infinite sources of unlabeled data, which makes effective unsupervised learning an enticing research direction. Here we aim to develop a technique that enables models to generalize better by incorporating auxiliary self-supervised auditory tasks during model training BID4 ).Our main contributions in this paper are twofold: the successful identification of appropriate selfsupervised audio-related tasks and the demonstration that they can be trained jointly with supervised tasks in order to significantly improve performance. We also show how to use WaveNet as a general feature extractor capable of providing rich audio representations using raw waveform data as input. We hypothesize that by learning multi-scale hierarchical representations from raw audio, WaveNetbased models are capable of adapting to subtle variations within tasks in an efficient and robust manner. We explore this framework on three supervised classification tasks -audio tagging, speaker identification and speech command recognition -and demonstrate that one can leverage unlabeled data to improve performance on each task. We further show that these pair well with more common data augmentation techniques, and that our proposed self-supervised tasks can also be used as a pre-training stage to provide performance improvements through transfer learning. These authors contributed equally to this work. Prevailing wisdom suggests that a single model can only learn multiple tasks if they are related in some way with some underlying structure common to them BID3 ). Such structure has been described for decades in the literature on sensory environments, with Gabor filters and gammatone filters underlying much of visual and auditory processing, respectively BID14 ). Perhaps models trained to accomplish many tasks might be able to synergize to uncover this underlying structure, enabling better single-task performance with smaller amounts of data per-task. We follow a relatively common approach to multitask learning aimed at learning a single non-trivial general-purpose representation BID2 ). Examples of other intriguing approaches can be found in BID11 BID19 ).Much as shared representations allow models to pool data from different datasets, the problem persists that the cleanly labeled datasets that have permitted numerous breakthroughs in deep learning are painstaking to come by. One promising solution to label scarcity uses self-supervised learning to take advantage of unlabeled data. Self-supervised learning has shown promising in the visual domain, leveraging unlabeled data using tasks like inpainting for image completion BID13; BID18 ), image colorization BID7; BID22 ), and motion segmentation BID17 ). Despite these efforts, little previous work has taken advantage of self-supervision in the audio domain. We implemented an end-to-end audio processing network that finds a common embedding of the acoustic waveform within a "trunk" network modeled after the WaveNet architecture BID20 ). The embedding is then processed by simple, independent, task-specific "head" networks. The trunk and head networks are trained jointly for each experiment described below. Our experiments consist primarily of models in which a single supervised "main" task is trained jointly with 0 to 3 self-supervised "auxiliary" tasks. Briefly (see appendix for details), our WaveNet trunk consists of 3 blocks of 6 dilation stacks each. Each dilation stack is comprised of a gate and filter module, with 64 convolutional units per module. The outputs from the filter and gate modules are (elementwise) multiplied and then summed with the input to the stack. These choices yield a WaveNet trunk with an effective receptive field length of 1 + 3(2 6 − 1) = 190 samples or approximately 12 ms. We tested our setup on three distinct supervised tasks: audio tagging, speaker identification and speech command recognition. Each is trained using a separate labeled dataset along with up to three self-supervised tasks trained with unlabeled data. Our description of the tasks is necessarily brief, with details relegated to the appendix. The audio tagging task is trained on the FSDKaggle2018 dataset collected through Freesound. This dataset contains a total of 11,073 files provided as uncompressed PCM 16 bit, 44.1 kHz, monaural audio which is further subdivided into a training set and a test set. Before being fed to the network, each audio segment is first cropped to 2 seconds and padded with zeros if the source clip is too short. Since the WaveNet trunk produces embeddings with a temporal structure, this task averages the output across time to produce a single output vector for the entire audio sequence, which in turn feeds into a single fully-connected layer with 512 units and ReLU nonlinearity, followed by a softmax output layer. Training is done by minimizing the cross entropy between the softmax outputs and one-hot encoded classification labels. The speaker identification task is trained on the VoxCeleb-1 dataset BID12 ) which has 336 hours of data from 1251 speakers. Individual clips are sourced from interviews with celebrities in a variety of different settings. Data from each individual is sourced from multiple interviews and one interview is held-out to produce a test set with 15 hours of data. Before being fed to the network, each audio segment is first cropped to 2 seconds in duration. Given the large variations in the audio quality of the samples in this dataset, we found it necessary to also normalize the clips and apply a pre-emphasis filter. This task's head architecture features a global average pooling layer, followed by 2-layer perceptron with 1024 units per layer, batch normalization and a ReLU nonlinearity. The output is then passed to a softmax layer and evaluated using a cross-entropy loss. The speech command recognition task is trained on the Speech Commands dataset BID21 ).The entire dataset consists of 65,000 utterances of 30 short words, formatted in one-second WAVE format files. There is a total of 12 categories; 10 words (yes, no, up, down, left, right, on, off, stop, go), with the rest classified as either unknown or silence. The speech command recognition head is a stack of three 1D convolutions. Between each convolutional layer we used batch normalization and dropout, followed by a ReLU nonlinearity. The three convolution layers have widths of 100, 50, and 25 and strides of 16, 8, and 4, respectively. The output is passed to a final softmax layer and evaluated using a cross-entropy loss. We selected next-step prediction, noise reduction, and upsampling for our self-supervised, auxiliary tasks. They are easily implemented and can be synergistically paired with our main (supervised) tasks. The self-supervised tasks were trained on both the main task's data and unlabeled data sampled from the 100-hour and 500-hour versions of the Librispeech dataset BID15 ). This dataset was only used to train the auxiliary tasks. All three auxiliary tasks share the same basic head architecture. They begin with two convolutional layers with 128 filters and ReLU nonlinearities and a final linear convolutional layer with 1 output unit feeding into a regression-type loss function (see appendix for details). Our primary goal was to develop a multitask framework which is completely generic for audio, making it prudent to work with waveform inputs as opposed to, say, "high level" feature representations like spectrograms. While convolutional architectures trained on spectral/cepstral representations of audio can indeed give better classification performance than models trained directly on raw waveforms, they significantly restrict the range of audio processing tasks which they can perform. Thus, state-of-the-art baseline models for different tasks may vary wildly in their network architectures, subsequently limiting the amount of information that can be gained from a smaller pool of potential self-supervised tasks. If the goal is to understand the interaction between the learning dynamics of disparate tasks, then the focus should be on models which make the fewest assumptions about the representation of inputs. As such, we emphasize improvements in performance afforded by multitask learning relative to a single task baseline trained on raw audio. Closing the performance gap between models trained using spectral representations (e.g. BID8 ; BID12) and those trained on waveforms is left to future work. Joint training with three self-supervised tasks proved beneficial for each of our three supervised tasks TAB0. For the audio tagging task, multitask training improved MAP@3 score by.019 and top-1 classification rate by 1.62%, simply by including additional unsupervised tasks without increasing training data. Since the auxiliary tasks can be trained with unlabeled data, we gradually incorporated larger versions of Librispeech into our training regimen to investigate the effects of self-supervision. With each increase in unlabeled dataset size, we saw a further improvement on both performance metrics, with a MAP@3 increase of up to.056 with an additional 500 hours of unlabeled data. Using the same setup, but swapping the audio tagging task with either the speech command classification or the speaker identification task showed a similar, though more measured, trend with increasing amounts of unlabeled data. Speech command classification went from 93.05% in the baseline model to 93.78% when trained with an additional 500 hours of unlabeled data. Speaker identification on the VoxCeleb dataset was a much more challenging task for the network overall. There, top-5 classification performance peaked at 75.22%, up from the baseline performance of 73.81%. The above show that multitask learning can improve the performance of any of our supervised tasks without any additional labeled data. To get an idea of the significance of the observed effects, we decided to compare the above with another common technique for improving label efficiency: data augmentation. We trained a single task model on audio tagging with two different kinds of data augmentation: pitch shifting and additive noise (with SNRs of 10 to 15 dB). We found that pitch-shift augmentation produced an increase in MAP@3 of.066, comparable to our largest multitask benefits TAB1. Noise augmentation showed a somewhat smaller MAP@3 increase of.024. Interestingly, the performance gains from augmenting with noisy data are similar to those obtained by training the main task jointly with a self-supervised noise-reduction task. Finally, training with both pitch-shift augmentation and additional self-supervised tasks yielded a MAP@3 increase of.089 -our highest performance from any experiment -suggesting that both methods for improving label efficiency are complementary. In computer vision, the scope of transfer learning has been enlarged to include knowledge transfer from self-supervised tasks trained on unlabeled data to supervised tasks BID4 ). This inspired us to reconsider our multitask learning approach from a transfer learning perspective. In this variant of transfer learning, we jointly "pre-train" our three self-supervised tasks on purely unlabeled data to convergence. We follow this up with a fine-tuning stage, using a much smaller quantity of labeled data, to train a supervised task. We carried out transfer learning experiments on the same trio of tasks tackled above in our multitask learning experiments. The (see TAB2) favor transfer learning over simultaneously training all tasks together. The present work developed the following theme: faced with training an audio task on limited quantities of labeled data, one can expect performance gains by jointly training the supervised task together with multiple self-supervised tasks using a WaveNet-based model operating directly on raw audio waveforms. We have shown that the improved performance on the supervised tasks scales with the quantity of unlabeled data and can be used to supplement existing data augmentation schemes. Predicated on the performance gains observed on three fairly distinct audio classification tasks, we expect our approach to generalize to a broad range of supervised audio tasks. Our methodology and suggest many interesting directions for further development. Is there a limit on the number of auxiliary tasks that a single model at fixed capacity can benefit from, and can one place bounds on the expected improvement in performance? Intuitively, we expect that when our multitasking model learns to simultaneously forecast frames of audio, remove noise from the audio and perform upsampling, it must have formed a representation of the audio. What is this representation? Can it be extracted or distilled? A proper exploration of these questions should enable us to handle a broader range of auditory tasks. Although audio tag classification does not require the fine temporal resolution found in raw audio waveforms, our chosen auxiliary tasks (or any arbitrary auditory task for which we may desire our model to be sufficient) require higher temporal resolutions. To satisfy this, we chose to build our model following the WaveNet architecture (van den).WaveNet models are autoregressive networks capable of processing high temporal resolution raw audio signals. Models from this class are ideal in cases where the complete sequence of input samples is readily available. WaveNet models employ causal dilated convolutions to process sequential inputs in parallel, making these architectures faster to train compared to RNNs which can only be updated sequentially. ing small, task-specific neural networks built atop a task-agnostic trunk. The trunk architecture principally follows the structure of WaveNet, with several blocks of stacked, dilated, and causal convolutions between every convolution layer. Outputs from the trunk are fed into task-specific heads (details in Section 6.1).As shown Figure 6.1, our WaveNet trunk is composed of N blocks, where each block consists of S dilated causal convolution layers, with dilation factors increasing from 1 to 2 S − 1, residual connections and saturating nonlinearities. We label the blocks using b = 1, · · ·, N. We use indices ∈ [1 + (b − 1)S, bS] to label layers in block b. Each layer,, of the WaveNet trunk consists of a "residual atom" which involves two computations, labeled as "Filter" and "Gate" in the figure. Each residual atom computation produces a hidden state vector h and a layer output x defined via DISPLAYFORM0 where denotes element-wise products, represents the regular convolution operation, denotes dilated convolutions with a dilation factor of 2 mod bS if is a layer in block b+1, σ denotes the sigmoid function and W The first (= 0) layer -represented as the initial stage marked "1×1 Conv" in Figure 6.1 -applies causal convolutions to the raw audio waveforms X = (X 1, X 2, · · ·, X T), sampled at 16 kHz, to produce an output DISPLAYFORM1 Given the structure of the trunk laid out above, any given block b has an effective receptive field of 1 + b(2 S − 1). Thus the total effective receptive field of our trunk is τ = 1+N (2 S −1). Following an extensive hyperpameter search over various configurations, we settled on N = 3 blocks comprised of S = 6 layers each for our experiments. Thus our trunk has a total receptive field of τ = 190, which corresponds to about 12 milliseconds of audio sampled at 16kHz. As indicated above, each task-specific head is a simple neural network whose input data is first constrained to pass through a trunk that it shares with other tasks. Each head is free to process this input to its advantage, independent of the other heads. Each task also specifies its own objective function, as well as a task-specific optimizer, with customized learning rates and annealing schedules, if necessary. We arbitrarily designate supervised tasks as the primary tasks and refer to any self-supervised tasks as auxiliary tasks. In the experiments reported below, we used "audio tagging" as the primary supervised classification task and "next-step prediction", "noise reduction" and "upsampling" as auxiliary tasks training on various amounts of unlabeled data. The parameters used for each of the task specific heads can be found in TAB3 of the accompanying supplement to this paper. Figure 2: The head architectures were designed to be simple, using only as few layers as necessary to solve the task. Simpler head architectures force the shared trunk to learn a representation suitable for multiple audio tasks. The next-step prediction task can be succinctly formalized as follows: given a sequence {x t−τ +1, · · ·, x t} of frames of an audio waveform, predict the next value x t+1 in the sequence. This prescription allows one to cheaply obtain arbitrarily large training datasets from an essentially unlimited pool of unlabeled audio data. Our next-step prediction head is a 2-layer stack of 1 × 1 convolutional layers with ReLU nonlinearities in all but the last layer. The first layer contains 128 units, while the second contains a single output unit. The head takes in τ frames of data from the trunk, where τ is the trunk's effective receptive field, and produces an output which represents the model's prediction for the next frame of audio in the sequence. The next-step head treats this as a regression problem, using the mean squared error of the difference between predicted values and actual values as a loss function, i.e. given inputs {x t−τ +1, · · ·, x t}, the head produces an output y t from which we compute a loss L MSE (t) = (y t − x t+1) 2 and then aggregate over the frames to get the total loss. We would like to note that the original WaveNet implementation treated next-step prediction as a classification problem, instead predicting the bin-index of the audio following a µ-law transform. We found that treating the task as a regression problem worked better in multitask situations but make no claims on the universality of this choice. In defining the noise reduction task, we adopt the common approach of treating noise as an additive random process on top of the true signal: if {x t} denotes the clean raw audio waveform, we obtain the noisy version viax t:= x t + ξ t where ξ t an arbitrary noise process. For the denoising task, the model is trained to predict the clean sample, x t, given a window x t− 1 2 (τ −1), · · ·,x t+ 1 2 (τ −1) of noisy samples. Formally speaking, the formulation of the next-step prediction and denoising tasks are nearly identical, so it should not be surprising to find that models with similar structures are well-adapted to solving either task. Thus, our noise reduction head has a structure similar to the next-step head. It is trained to minimize a smoothed L1 loss between the clean and noisy versions of the waveform inputs, i.e. for each frame t, the head produces an outputŷ t, and we compute the loss DISPLAYFORM0 and then aggregate over frames to obtain the total loss. We used the smooth L1 loss because it provided a more stable convergence for the denoising task than mean squared error. In the same spirit as the denoising task, one can easily create an unsupervised upsampling task by simply downsampling the audio source. The downsampled signal serves as input data while the original source serves as the target. Upsampling is an analog of the "super-resolution" task in computer vision. For the upsampling task, the original audio was first downsampled to 4 kHz using the resample method in the librosa python package BID10 ). To keep the network operating at the same time scale for all auxiliary tasks, we repeated every time-point of the resampled signal 4 times so as to mimic the original signal's 16 kHz sample rate. The job of the network is then to infer the high frequency information lost during the transform. Again, given the formal similarity of the upsampling task to the next-step prediction and noisereduction tasks, we used an upsampling head with a structure virtually identical to those described above. As with the denoising task, we used a smooth L1 loss function (see eqn. above) to compare the estimated upsampled audio with the original. We trained the model using raw audio waveform inputs taken from the FSDKaggle2018 and Librispeech datasets. All code for the experiments described here was written in the PyTorch framework BID16. All audio samples were first cropped to two seconds in duration and downsampled to 16 kHz. To normalize for the variation in onset times for different utterances, the 2 seconds were randomly selected from the original clip. Samples shorter than 2 seconds were zero padded. We then scaled the inputs to lie in the interval [−1, 1]. The noise-reduction task required noisy inputs which we obtained by adding noise sampled from ChiME3 datasets BID1 at a randomly chosen SNR from 10dB to 15dB. The noise types include booth (BTH), on the bus (BUS), cafe (CAF), pedestrian area (PED), and street junction (STR)). Starting with the main task, we first performed a hyperparameter search over the number of blocks in the trunk, the number of layers per block, the number of layers and units of the main task head, and the learning rate. We tried several values for the number of blocks in the trunk, ranging from 2 to 5. We also varied the number of dilated convolution layers in each block from 3 to 8. We found that the performance and training characteristics of the network were largely unaffected by the exact architecture specifications, though learning rate was often important. We then searched over the depth and width of each auxiliary task head, as well as the learning rate for the head. These searches were done by pairing each task individually with the main task. The final choice of hyper-parameters was made by picking values which gave the best possible performance on both the main task and the auxiliary tasks, heuristically favoring performance on the main task. We jointly trained the model on all tasks simultaneously by performing a forward pass for each task, computing the loss function for each task, and then calculating the gradients based on a weighted sum of the losses, viz. L total = i α i L i, where the sum runs over all the tasks. We used a uniform weighting strategy in our current experiments. More advanced weighting strategies showed no benefit for the tagging task. We used the "Adam" optimizer BID6 with parameters β 0 = 0.9, β 1 = 0.99, ε = 10 −8. The learning rate was decayed by a factor of.95 every 5 epochs, as this was found to improve convergence. We used a batch size of 48 across all experiments, since it was the largest batch size permissible by the computational resources available to us. Adding the noise reduction and upsampling tasks required a separate forward propagation of the noisy and downsampled audio, respectively. Exact values for all important parameters of the model can be found in TAB3.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Label-efficient audio classification via multi-task learning and self-supervision

Represent the TLDR for finding the corresponding scientific article

We present a new approach, SNIP, that is simple, versatile and interpretable; it prunes irrelevant connections for a given task at single-shot prior to training and is applicable to a variety of neural network models without modifications.