Question: Represent the text to find a very short summary (1-2 sentences max).

Question: Knowledge bases (KB) are often represented as a collection of facts in the form (HEAD, PREDICATE, TAIL), where HEAD and TAIL are entities while PREDICATE is a binary relationship that links the two. It is a well-known fact that knowledge bases are far from complete, and hence the plethora of research on KB completion methods, specifically on link prediction. However, though frequently ignored, these repositories also contain numerical facts. Numerical facts link entities to numerical values via numerical predicates; e.g., (PARIS, LATITUDE, 48.8). Likewise, numerical facts also suffer from the incompleteness problem. To address this issue, we introduce the numerical attribute prediction problem. This problem involves a new type of query where the relationship is a numerical predicate. Consequently, and contrary to link prediction, the answer to this query is a numerical value. We argue that the numerical values associated with entities explain, to some extent, the relational structure of the knowledge base. Therefore, we leverage knowledge base embedding methods to learn representations that are useful predictors for the numerical attributes. An extensive set of experiments on benchmark versions of FREEBASE and YAGO show that our approaches largely outperform sensible baselines. We make the datasets available under a permissive BSD-3 license. Knowledge Bases (KBs) are playing an increasingly important role in a number of AI applications. KBs can be seen as a collection of facts or triples of the form (head, predicate, tail), denoted as (h, p, t), where head and tail correspond to entities and predicate corresponds to a relationship that holds between these two entities. This structured information is easily accessible by AI systems to enhance their performance. A variety of AI applications such as recommender systems, natural language chatbots or question answering models, have benefited from the rich structural information archived in these repositories. This is because much of human knowledge can be expressed with one or more conjunctions of knowledge facts. However, KBs' capabilities are limited due to their incompleteness 1. Consequently there has been a flurry of research on knowledge base completion methods in recent years. Relationship extraction BID27 (i.e., classification of semantic relationship mentions), knowledge graph matching BID32 BID12 (i.e., alignment and integration of entities and predicates across KBs), or search-based question-answering BID36 (i.e., queries issued to a web search engine) are a few different ways to address the incompleteness problem. However, the literature on the so-called link prediction methods BID22 has received more attention in the last few years in comparison to the aforementioned approaches. Contrary to other solutions, link prediction methods aim to find missing links between entities exclusively based on the existing information contained in the KB. This is achieved by ranking entities that are answer candidates for the query. The queries these methods typically address are of the form (USA, /location/contains, ?), or (Madrid, /location/capitalOf, ?), whereas the missing element -represented by a question mark-is an entity contained in the KB.Many link prediction methods only harness feature types learned from the rich relational information contained in the KB to infer new links, and only very recently , ] numerical attributes have been integrated along with other feature types to improve link prediction performance. Similarly, numerical information is also represented as facts such as (Berlin, /location/latitude, 52.31) or (Albert Einstein, /person/birth year, 1879). However, as shown in BID5 ] the application of numerical attributes is limited because of the same incompleteness problem: Many entities are missing numerical attribute values they are expected to possess. For example, entities that represent locations should have numerical information regarding latitude, longitude or area, among others; whereas for entities representing people, numerical predicates such as the birth year, weight or height would be more appropriate. In this work we focus on the problem of completing queries where the relationship is a numerical predicate. Consequently, the answer to this new type of query is a numerical value. This is contrary to the link prediction problem, wherein the answer to a query is always an element of a closed vocabulary. Examples of queries addressed in this paper are (Apple Inc., revenue, ?) or (California, average salary, ?). While one can interpret link prediction as a classification/ranking problem, this is rather a regression problem. The main contributions of this paper are:• We introduce the problem of predicting the value of entities' numerical attributes in KBs. For the sake of simplicity we term this as'numerical attribute prediction problem'. To our knowledge, this is the first time this problem is addressed in the literature.• We create benchmark datasets for this problem. We use well-known subsets of Freebase and Yago as the blueprints for creating these benchmarks. We also create versions of these datasets for different percentages of sparsity by artificially removing facts that involve numerical predicates. All these benchmark datasets will be made publicly available.• We propose two meaningful baselines for this problem. These baselines are inspired by previous work done in the node classification and the imputation literature.• We propose supervised and semi-supervised approaches to this problem. The semisupervised approaches significantly outperform the baselines in all datasets and conditions. The paper is organized as follows: We discuss the related work in Section 2. Afterwards we formalize the problem of predicting numerical values for entities' numerical attributes in KBs in Section 3. We describe our approaches to this problem, as well as the two baselines. Section 4 reports the experimental setting followed by an extensive set of experiments on different datasets with different degrees of sparsity in Section 5. Finally, we summarize the of our study in Section 6. There is an extensive body of work on link prediction BID2 BID37 BID22 BID34. Logical approaches [] operate on a set of logical rules that are usually handcrafted and/or mined. These logical formulas are evaluated between entity pairs to generate feature representations which are then used for a downstream machine learning model. On the other hand, KB embedding methods BID22 learn feature representations -embeddings-for all elements in a KG by optimizing a designed scoring function. Given a fact, these scoring functions output a score that relates to the likelihood of that fact being true. A popular and successful instance of KB embedding method is TransE BID2, where predicates are modeled as translations in the entity embedding space. Much less work has been done on entity-type classification BID19,. This problem is inherently related to link prediction, since it amounts to complete queries of the form (head, typeOf, ?), where the question mark corresponds to a certain entity type (e.g. location, artist, ...). Therefore, link prediction and entity-type classification share certain similarities with the numerical attribute prediction problem. Most importantly, they all make use of the relational information for KB completion, one way or another. However, there is a crucial difference between link prediction and numerical attribute prediction. In the former, a query can be completed with one or several elements contained in a relatively small vocabulary, whereas in the later the answer may (potentially) take an infinite number of real values. There is another line of research related to our work, namely value imputation BID28. In statistics, imputation is the process of replacing missing data with substituted values. In the simplest case, one can replace the missing values of a variable by the mean of all existing values of that variable. This technique is called mean imputation. It preserves the mean of the variable, but alters the underlying variable distribution to be more peaked at the mean BID0. However, it is the most commonly practiced approach for value imputation BID29, and it has been shown to be competitive for a number of downstream tasks BID1 , ]. Another popular approach is called regression imputation, where the missing values of a variable are estimated by a regression model from the observed values of other variables. There is some work on using text for predicting numerical attributes of entities such as BID3 , ]. BID9 uses Word2Vec embeddings of named entities as inputs to a number of regression models. Similar to us, they aim to predict numerical attributes of knowledge base entities. Different to us, they leverage text information to do so. This difference is important, because we do not assume the existence of information other than the graph structure. Our problem is general enough to address knowledge bases where entities names are unknown or anonymized (e.g. medical knowledge bases).To our knowledge there is no existing work in the value imputation literature that attempts to fill missing values in KBs while taking advantage of the structural information provided by the KB. A knowledge base, KB, is denoted as G = (E, P), where E is a set of entities and, P is a set of relation types or predicates. This standard definition can be found in many papers in the link prediction literature [, Garcia-Duran and . A KB is a collection of facts (or standard facts) (h, p, t) where p ∈ P and h, t ∈ E. We now define a knowledge base enriched with numerical attributes as G N A = (G, A, N). Entities in G are associated with numerical values N via numerical predicates A. This information can be expressed as a collection of numerical facts (h, a, t) where a ∈ A, h ∈ E and t ∈ N. In the paper we interchangeably use the term'numerical predicate' with'numerical attribute'. The numerical attribute prediction problem seeks the most probable completion of a fact (h, a, ?), where h ∈ E, a ∈ A and? ∈ N.We refer to the set of entities for which the value of the numerical attribute a is known as E a ⊆ E. Let e be an entity with numerical attribute a, then we denote the known numerical value for attribute a as n a e. The goal is to learn a function f: E →, denotes the set of reals. One can omit the relational information given by the graph G and apply a value imputation method to fill missing values. However, it is intuitive to assume the existence of an underlying generative model that (partially) determines the relational structure of the KB based on the values of the entities' numerical attributes. For instance, two entities are likely linked via the relationship /location/contains if they have similar latitude and longitude; or two highly connected entities that correspond to people are likely to have similar birth years. If this assumption is true, then a model that exploits the graph structure information is likely to outperform simple value imputation methods. Nevertheless, while this may be true for a number of numerical attributes, for others the graph structure may introduce noise or, in the best case, be irrelevant. Inspired by previous work in the value imputation and the node classification literature, we propose the following baselines. A simple and natural baseline is simply using the sample mean of the attribute specific training data as a predictor for missing values. This is known as mean imputation BID29. At test time, given an entity e for which we aim to predict the value of numerical predicate a, denoted asn a e, this baseline simply assigns the sample mean of all known entities possessing the same numerical attribute (E a). This is formally described below.n DISPLAYFORM0 where f is the sample mean. We term this model as Global because it harnesses global information from the entire attribute specific training set. In this work we use the root mean square error (RMSE) and the mean absolute error (MAE) as evaluation metrics. While the sample mean is the best estimator for the former, the sample median is the optimum for the latter BID20. Consequently, in the experimental section we use median imputation when reporting the MAE and mean imputation when reporting on RMSE metrics. Median imputation is obtained by simply replacing the sample average by the median in Eq.. Our second baseline takes into account that entities are interconnected through a relational graph structure. Thus it is natural to define a baseline that exploits the neighborhood or local graph structure. The weighted-vote relational neighbor BID14 ] is a relational classifier often used as a benchmark in the node classification literature. It estimates the class label of a node as a weighted average of its neighbors' class labels. Despite its simplicity, it is shown to be competitive BID24 and is advocated as a sensible relational classification baseline BID15.Inspired by such work, we propose an adaptation for our setting and problem. For a numerical attribute a, this baseline estimates a value for the entity e as the average of its neighbors' attribute values for that numerical attribute. Here, the neighborhood of a node e, denoted N e, is defined as the set of nodes that are connected to e through any relation type. The baseline is formalized as followŝ DISPLAYFORM0 where, as before, f is either the sample mean or the sample median depending on the evaluation metric reported. We term this model as Local because it uses the local neighborhood information for prediction. In the case where E a ∩ N e = ∅, we make use of the so-called Global baseline to make a prediction. We leverage KB embedding methods to learn feature representations -embeddings-of entities that (ideally) are predictive for the different numerical attributes. As we argued before, this is only true if the entities' numerical attributes determine, to a certain extent, the existence of a certain relation type between two entities. We first learn knowledge base embeddings, and in a second step we use these embeddings, along with the numerical facts, to train downstream machine learning models for predicting numerical attributes. This pipeline is reminiscent of recent work BID25 in the node classification literature. While there is an extent literature on KG embedding methods BID21 BID2 BID22, recent work BID10 shows that well-tuned "simple" scoring functions BID37 are very hard to beat. Likewise BID6 shows that TransE BID2 performs similarly or even better than many of its variants, such as TransH BID35 or TransR BID13.Due to its simplicity and good performance in related problems, we choose TransE to illustrate the generic principles behind our models. Note, however, that the methodology described is agnostic to the chosen KG embedding method. The probability for a fact DISPLAYFORM0 c g(c|θ)), where c indexes all possible triples, and θ all learnable parameters of TransE, whose scoring function g is g(d | θ) = ||h + p − t|| 2. We use bold letters h, p, t ∈ d to denote the corresponding d-dimensional feature representations of h, p, t, respectively. Note that this formulation is impractical because the cost of computing all possible triples is unfeasible. Instead, for each triple d = (h, p, t) ∈ G we generate a set of N triples (h, p, t) by sampling N entities t uniformly at random from the set of all entities. This process, which is termed as negative sampling, is repeated for the head of the triple. For a given set of facts D that are part of the KB G, the logarithmic loss is defined as DISPLAYFORM1 All parameters θ are learned for minimizing L G with stochastic gradient descent. Once the representation learning phase is finished we evaluate two different approaches that utilize these embeddings for addressing the numerical attribute prediction problem. In the simplest case, for each numerical attribute we use the learned feature representations as input to a regression model to predict the corresponding numerical attribute. For numerical attribute a the loss function is given by DISPLAYFORM0 where f ϑ a refers to the regression function for numerical attribute a, ϑ a refers to the learnable parameters of f ϑ a, and λ a is the regularization hyper-parameter. In this work we use a linear regression model: f ϑ a (e) = e T w a + b a, where w a ∈ d is the weight vector and b a is the corresponding bias term. At test time, given a query related to a certain numerical attribute a and a certain entity e, the prediction is computed by applying the corresponding linear regression model:n a e = f ϑ a (e). We refer to this approach as Lr. Previously we defined E a as the set of entities with known numerical attribute a. Similarly, we define Q a as the set of entities with missing values for numerical attribute a. We consider numerical attribute values as labels, and, consequently, we can think of E a and Q a as the set of labeled and unlabeled nodes, respectively. Therefore, semi-supervised learning is a natural choice because it also uses unlabeled data to infer values of numerical attributes. Label propagation (LP) BID13 , Fujiwara and BID4 has been proven to be an effective semi-supervised learning algorithm for classification problems. The key assumption of label propagation, and in general most semi-supervised learning algorithms, is similar to ours: Data points close to each other are likely to have the same label -numerical attribute values in our case. We aim to propagate numerical attribute information across the graph using LP. For numerical attribute a, we use the learned representations {e} e∈E a ∪Q a to induce a k-nearest neighbor graph (kNN) using euclidean distance. This graph is characterized by an adjacency matrix A ∈ N ×N, where N = |E a | + |Q a |. The edge weights of the adjacency matrix represent similarities between the connected entities, which are computed according to a similarity metric ρ -in this work we use a radial basis function kernel 2.We then compute the transition matrix T by row-wise normalizing the matrix A. Without loss of generality, we arrange labeled and unlabeled data so that T can be decomposed as DISPLAYFORM0 2. ρ(x, y) = exp(− ||x − y|| The transition matrix T (illustrated in FIG2) can be iteratively used to propagate numerical information across the graph until a stopping criterion is reached. Alternatively, this problem can be solved in a closed form: DISPLAYFORM1 where n a E a ∈ |E a | is a vector that contains all values of numerical attribute a for labeled nodes. Similarly,n a Q a ∈ |Q a | is a vector that contains all predicted values of numerical attribute a for unlabeled nodes. We refer to the matrix M a in Section 5.1. We term this as Numerical Attribute Propagation (or Nap).Related work BID18 uses label propagation to perform link prediction in web ontologies by casting it as a binary classification problem, where the similarity graph is built based on homophilic relationships. In the two aforementioned solutions, we fully rely on the feature representations learned by, in this case, TransE to be meaningful with respect to the numerical attributes we aim to predict. This relates to our initial assumption that the relational structure of a KB can be explained, to some extent, by the numerical attributes of the entities. However, there might be cases where the values taken by entities for a certain numerical attribute do not fully relate to the relational structure of the KB.Motivated by this consideration, we set out to answer the question: Can these models benefit from learning feature representations incorporating, beside the graph structure, numerical attribute information?To answer this question we incorporate the learning objective of Eq. into the learning objective of TransE (Eq.): DISPLAYFORM0 where α weights the importance of the linear regression objectives. All parameters are learned using stochastic gradient descent. We term these embeddings as TransE++, which, contrary to TransE, are also learned with numerical facts. While Nap and Lr use TransE feature representations, their counterparts Nap++ and Lr++ leverage TransE++ embeddings. Different numerical attributes exhibit different scales of attribute values. For example'person.height' ranges from 1 to 2 meters while'location.area' scales from several hundred to even millions of kilometers. Bigger scales lead to larger prediction errors during training which in turn affect the back-propagation gradients used for learning the TransE++ embeddings. To alleviate this problem, we normalize each numerical attribute to zero-mean and unit-variance. We also experimented with min-max scaling, however it gave worse performance compared to standard scaling. Scaling numerical attribute values remains an interesting challenge. For Lr++ we do not directly use the regression models learned during opimizing the TransE++'s learning objective (Eq. FORMULA7). Instead we use the learned TransE++ embeddings to train a new regression model L a R for each numerical attribute a ∈ A. This is because of the computational difficulty in tuning hyper parameter λ a for each numerical attribute while learning TransE++, which we found important to obtain good performance. Note that the hyper parameter space grows exponentially with the number of attributes |A|. For this reason, we set λ a = λ = 0 in the regression objectives in Eq. for learning TransE++ embeddings (first step). For the final regression models (second step) we do tune λ a s (independently), which, though suboptimal, facilitates their tuning. The proposed methods are evaluated by their ability to answer completion queries of the form (h, a, ?), where h ∈ E, a ∈ A and? ∈ N. We evaluate the baselines and our models on two benchmark datasets: FB15K-237 BID33 and YAGO15K. While for the former, numerical attributes were introduced in BID5, for the later we obtained this information from dumps found online on YAGO's website 3.The FB15K-237 dataset contains a total of 29,395 numerical facts divided in 116 different numerical predicates. We evaluate our models on the top 10 numerical attributes ranked by the number of data samples. This reduces the dataset to 22,929 samples. We split these numerical facts into training, validation and test in the proportion of 80/10/10%, respectively. All other facts from FB15K-237 whose predicate belongs to P are used as training data, which amounts to 310,116 facts. Thus we only evaluate our approaches on their ability to answer queries whose answer is a numerical value. The YAGO15K dataset contains 23,520 numerical facts divided in 7 different attributes. Similarly, we split these numerical facts into training, validation and test in the same proportion. We use all other 122,886 facts from this dataset for learning knowledge base embeddings. A summary of the datasets can be found in Table 1. All the splits of both datasets used in this work will be made publicly available to facilitate future comparisons. We compare performance across methods using two evaluation metrics -MAE and RMSE. These are standard metrics used in regression problems and were also used in a related study of predicting numerics for language models BID31. DISPLAYFORM0 For TransE and TransE++ we fix the embedding dimension d to 100. After some preliminary experiments, the weight α of TransE++ was fixed to 1. We used Adam BID11 ] to learn the parameters in a mini-batch setting with a learning rate of 0.001. We fixed the number of epochs to 100 and the mini-batch size to 256. The parameter N of the negative sampling was set to 50. Within a batch, the number of data points for each of the TransE++'s regression objectives is proportional to the frequency of each of the numerical predicates in the training set. In all cases, the parameters were initialized following BID8.We used the Scikit-learn BID23 implementation of ridge regression for the approaches Lr and Lr++. The regularization term λ a is tuned using the values [0, 0.1, 1, 10, 100].For Nap and Nap++ the number of neighbors (k) of the kNN graph is validated among; and the σ of the RBF kernel is validated among [0.25, 0.5, 1, 10].All of the above is validated for each numerical predicate and evaluation metric. The objectives of this section is twofold: First we investigate the performance of our approaches (Lp and Nap, and their variants) with respect to the baselines. And second we experimentally check how robust these methods are for different degrees of sparsity in the training data. TAB2 detail the performance of the baselines and our approaches on FB15K-237. For each numerical attribute we always indicate in bold font the best performing method, which happens to be either Nap++ or Nap most of the time. Interestingly enough, from TAB2 we observe that for the numerical attributes'location.area' and'population.number' Global largely outperforms Local. This seems to indicate that the relational structure of this data set does not relate to these two numerical predicates. Overall, predictions for all other numerical attributes tend to benefit from the local information given by the entities' neighborhood. In comparing Tables 2 and 3, we note that Local is very competitive in Table 3: Performance of Lr-and Nap-based models on FB15K-237.regard to the numerical attributes'latitude' and'longitude'. This can be explained by the presence of predicates such as'location.adjoins' or'location.contains' in the relational structure of the graph. Similarly, entities' neighborhoods are useful for predicting'date of birth' or'date of death' because (some of the) surrounding entities correspond to people who have similar birth or death dates. Interestingly, all our approaches beat both baselines in the numerical attribute'person.height mt', for which a priori one would not expect performance gains in learning from the graph structure. Overall, Lr++ and Nap++ outperform their counterparts Lr and Nap, respectively, for most numerical predicates. As we argued in Section 3.2.3 it is not feasible to validate the regularization term λ a for every numerical attribute while learning TransE++. We speculate that setting λ a = 0 while training TransE++ may explain why Lr++ and Nap++ do not always beat their counterparts. Table 4: Performance of Local and Nap++ on FB15K-237 for different degrees of sparsity, P r, on the numerical facts. Results are reported in terms of Mean Absolute Error (MAE).Another observation from Table 3 is that, in general, Nap-based models perform much better compared to Lr-based models. One can find a number of explanations to this. The obvious explanation is that the numerical attribute propagation approaches learn from labeled and unlabeled data, whereas the regression models only learn from labeled data. A second explanation is that whereas Nap's predictions are computed as a weighted average of observed numerical values, Lr's predictions are not bounded. This prevents Nap-based approaches from making large mistakes. On the other hand, for example, we observed non-plausible values (e.g. > 2020) predicted by the Lr-based models for the numerical attribute'date of birth'. We also experimented with non-linear regression models, but did not observe any performance improvement. Knowledge graphs are known to suffer from data sparsity due to missing facts. The same incompleteness is also true for numerical facts. Therefore it is crucial to study model performance under a sparse data regime. We generate data sparsity by artificially removing numerical facts from the training set while keeping the validation and test sets unchanged. We keep the underlying knowledge graph G unchanged because we aim to isolate the effect of numerical fact sparsity. In other words, only a number of numerical facts are removed from the training set. We retained a percentage P r of training numerical facts and ran Local and Nap++ with the same experimental set-up. We experimented with the following values of P r: %. We detail the of these experiments in Table 4. Note that the performance of Local degrades more rapidly compared to Nap++ as the sparsity increases. Even in high regimes of sparsity, Nap++'s performance is remarkably robust. TAB5 lists for Global and Local in YAGO15K. As for FB15K-237, Local outperforms Global for most of the numerical attributes. This reinforces our assumption that the numerical attributes explain, to some extent, relation structure between entities. Table 6 depicts the performance of Local, Nap and Nap++ under different degrees of sparsity in YAGO15K. In the light of these numbers, we can conclude that the Nap-based Table 6: Performance of Local, Nap and Nap++ on YAGO15K for different degrees of sparsity, P r, on the numerical facts. Results are reported in terms of Mean Absolute Error. models are more robust than Local to data sparsity. Nap++ achieves the best performance for most of the numerical attributes and degrees of sparsity. It performs remarkably well for the numerical attribute'happenedOnDate' in comparison to Nap. Across all values of P r, on average, NAP++ improves Nap's performance by 20 points (in mean absolute value) for'happenedOnDate'. We recognize that reporting model performance in absolute values complicates comparison since numerical attributes lie on different ranges of values. To have a better picture of performance gains we report percentage error reduction between Nap++ and the best performing baseline. For numerical attribute a, the percentage error reduction in MAE is computed as follows DISPLAYFORM0 One can compute the percentage error reduction in terms of RMSE in a similar manner. This is shown in TAB7 for P r = 100. We do not include'location.area' and'population.number', as previous experiments indicate that they do not relate to the graph structure of FB15K-237. Overall, Nap++ significantly outperforms baselines for almost all numerical attributes in both FB15K-237 and YAGO15K data sets. These demonstrate that the embeddings learned from the graph structure are useful predictors of entity numerical attributes. Table 8: Qualitative comparison between Nap and Nap++. The three first rows correspond to queries where the numerical attribute is'date of birth', whereas for the two last queries it is'date of death'. The actual value of labeled entities for the corresponding numerical attribute is shown in parenthesis. This last experimental section aims to provide some insights on the benefit of adding numerical information during the representation learning stage. Table 3 shows a noteworthy behavior of these methods with respect to the numerical attributes'date of birth' and'date of death'. While the performance of both approaches is comparable in terms of MAE, their RMSE largely differ. It is known that the mean absolute error is an evaluation metric more robust to outliers than the root mean squared error. We set out to inspect these outliers to shed light on the usefulness of incorporating numerical information in the embeddings. Nap-based models leverage these embeddings to build a similarity graph on which numerical information is propagated via Eq.. The ing predictions are the of multiplying the matrix M a by the observed numerical values. This matrix 5 determines which observed entities' numerical values to pay attention to. These attention values are different for Nap and Nap++ as the graph similarity is constructed with different embeddings. We qualitatively compare Nap and Nap++ based on a number of predictions computed in the test set. For each method we compare the two labeled entities they pay the most attention to. For the sake of simplicity we refer to these two entities as nearest neighbors. This is shown in Table 8. An interesting observation is that for NAP the two nearest neighbors are always entities topically similar to the entity in the query. On the other hand the nearest entities retrieved by NAP++ are more meaningful with respect to the queried numerical attribute. This is seen in the first query: (Alexander the Great 6, date of birth, ?). While Nap pays the most attention to topically similar entities, Nap++ puts a high attention on Julius Caesar, 7 which is more meaningful in regard to the date of birth. Nap++ uses Euclidean distance between vectors to build the k-nearest neighbor graph. Table 8 that subsets of entities latent factors could be encoding different relational and numerical information. For instance, a few dimensions of the entity embeddings encode location information, while others encode population information and so on. To exploit this we learned Mahalanobis metrics for capturing different entity similarities. We did this while learning knowledge base embeddings by using an additional nearest neighbor loss. It did slightly improve the performance for few attributes, but overall it did not make significant distance. We suggest that future work should address this research direction in greater depth. We introduce a novel problem, namely numerical attribute prediction in knowledge bases. Contrary to link prediction, the answer to this new query type is a numerical value, and not an element from a (small) closed vocabulary. Our premise to this problem is that the relational structure of a KB can be partially explained by the numerical attribute values associated with entities. This allows for leveraging KB embedding methods to learn representations that are useful predictors of numerical attributes. An extensive set of experiments validates our premise. Furthermore, we also show that learning KB representations enriched with numerical attribute information are helpful for addressing this task. Finally, we believe that this new problem introduced in the paper will spur interest and deeper investigation from the research community.5. Note that it is non-negative and is row normalized. 6. For all practical purposes he is deemed a philosopher in FB15K-237. 7. Julius Caesar belongs to profession Politician in FB15K-237

Represent this text to find a much much longer article about it with say hundreds-thousands of words:

Prediction of numerical attribute values associated with entities in knowledge bases.

Our aim in this paper is to propose a new approach for tackling the problem of transfer learning from labeled to unlabeled software projects in the context of SVD in order to resolve the mode collapsing problem faced in previous approaches.


Question: Represent the article for finding a single-sentence TLDR

Question: We propose procedures for evaluating and strengthening contextual embedding alignment and show that they are useful in analyzing and improving multilingual BERT. In particular, after our proposed alignment procedure, BERT exhibits significantly improved zero-shot performance on XNLI compared to the base model, remarkably matching pseudo-fully-supervised translate-train models for Bulgarian and Greek. Further, to measure the degree of alignment, we introduce a contextual version of word retrieval and show that it correlates well with downstream zero-shot transfer. Using this word retrieval task, we also analyze BERT and find that it exhibits systematic deficiencies, e.g. worse alignment for open-class parts-of-speech and word pairs written in different scripts, that are corrected by the alignment procedure. These support contextual alignment as a useful concept for understanding large multilingual pre-trained models. Figure 1: t-SNE visualization of the embedding space of multilingual BERT for English-German word pairs (left: pre-alignment, right: post-alignment). Each point is a different instance of the word in the Europarl corpus. This figure suggests that BERT begins already somewhat aligned out-of-the-box but becomes much more aligned after our proposed procedure. Embedding alignment was originally studied for word vectors with the goal of enabling cross-lingual transfer, where the embeddings for two languages are in alignment if word translations, e.g. cat and Katze, have similar representations (a;). Recently, large pretrained models have largely subsumed word vectors based on their accuracy on downstream tasks, partly due to the fact that their word representations are context-dependent, allowing them to more richly capture the meaning of a word (; ; ;). Therefore, with the same goal of cross-lingual transfer but for these more complex models, we might consider contextual embedding alignment, where we observe whether word pairs within parallel sentences, e.g. cat in "The cat sits" and Katze in "Die Katze sitzt," have similar representations. One model relevant to these questions is multilingual BERT, a version of BERT pre-trained on 104 languages that achieves remarkable transfer on downstream tasks. For example, after the model is fine-tuned on the English MultiNLI training set, it achieves 74.3% accuracy on the test set in Spanish, which is only 7.1% lower than the English accuracy (; b). Furthermore, while the model transfers better to languages similar to English, it still achieves reasonable accuracies even on languages with different scripts. However, given the way that multilingual BERT was pre-trained, it is unclear why we should expect such high zero-shot performance. Compared to monolingual BERT which exhibits no zero-shot transfer, multilingual BERT differs only in that during pre-training (i.e. masked word prediction), each batch contains sentences from all of the languages, and it uses a single shared vocabulary, formed by WordPiece on the concatenated monolingual corpora . Therefore, we might wonder: How can we better understand BERT's multilingualism? Can we further improve BERT's cross-lingual transfer? In this paper, we show that contextual embedding alignment is a useful concept for addressing these questions. First, we propose a contextual version of word retrieval to evaluate the degree of alignment, where a model is presented with two parallel corpora, and given a word within a sentence in one corpus, it must find the correct word and sentence in the other. Using this metric of alignment, we show that multilingual BERT achieves zero-shot transfer because its embeddings are partially aligned, as depicted in Figure 1, with the degree of alignment predicting the degree of downstream transfer. Next, using between 10K and 250K sentences per language from the Europarl corpus as parallel data , we propose a fine-tuning-based alignment procedure and show that it significantly improves BERT as a multilingual model. Specifically, on zero-shot XNLI, where the model is trained on English MultiNLI and tested on other languages (b), the aligned model improves accuracies by 2.78% on average over the base model, and it remarkably matches translate-train models for Bulgarian and Greek, which approximate the fully-supervised setting. To put our in the context of past work, we also use word retrieval to compare our finetuning procedure to two alternatives: fastText augmented with sentence and aligned using rotations (; Rücklé et al., 2018;), and BERT aligned using rotations (; ;). We find that when there are multiple occurences per word, fine-tuned BERT outperforms fastText, which outperforms rotation-aligned BERT. This supports the intuition that contextual alignment is more difficult than its non-contextual counterpart, given that a rotation, at least when applied naively, is no longer sufficient to produce strong alignments. In addition, when there is only one occurrence per word, fine-tuned BERT matches the performance of fastText. Given that context disambiguation is no longer necessary, this suggests that our fine-tuning procedure is able to align BERT at the type level to a degree that matches non-contextual approaches. Finally, we use the contextual word retrieval task to conduct finer-grained analysis of multilingual BERT, with the goal of better understanding its strengths and shortcomings. Specifically, we find that base BERT has trouble aligning open-class compared to closed-class parts-of-speech, as well as word pairs that have large differences in usage frequency, suggesting insight into the pre-training procedure that we explore in Section 5. Together, these experiments support contextual alignment as an important task that provides useful insight into large multilingual pre-trained models. Word vector alignment. There has been a long line of works that learn aligned word vectors from varying levels of supervision . One popular family of methods starts with word vectors learned independently for each language (using a method like skip-gram with negative sampling (b) ), and it learns a mapping from source language vectors to target language vectors with a bilingual dictionary as supervision (a; ;). When the mapping is constrained to be an orthogonal linear transformation, the optimal mapping that minimizes distances between word pairs can be solved in closed form . Alignment is evaluated using bilingual lexicon induction, so these papers also propose ways to mitigate the hubness problem in nearest neighbors, e.g. by using alternate similarity functions like CSLS (a). A recent set of works has also shown that the mapping can be learned with minimal to no supervision by starting with some minimal seed dictionary and alternating between learning the linear map and inducing the dictionary (; a; ; ;). Incorporating context into alignment. One key challenge in making alignment context aware is that the embeddings are now different across multiple occurrences of the same word. Past papers have handled this issue by removing context and aligning the "average sense" of a word. In one such study, learn a rotation to align contextual ELMo embeddings with the goal of improving zero-shot multilingual dependency parsing, and they handle context by taking the average embedding for a word in all of its contexts. In another paper, learn a rotation on sentence vectors, produced by taking the average word vector over the sentence, and they show that the ing alignment also works well for word-level tasks. In a contemporaneous work, align not only the word but also the context by learning a linear transformation using word-aligned parallel data to align multilingual BERT, with the goal of improving zero-shot dependency parsing numbers. In this paper, we similarly align not only the word but also the context, and we also depart from these past works by using more expressive alignment methods than rotation. Incorporating parallel texts into pre-training. Instead of performing alignment post-hoc, another line of works proposes contextual pre-training procedures that are more cross-lingually-aware. pre-train sentence embeddings using parallel texts by maximizing similarity between sentence pairs while minimizing similarity with negative examples. propose a cross-lingual pre-training objective that incorporates parallel data in addition to monolingual corpora, leading to improved downstream cross-lingual transfer. In contrast, our method uses less parallel data and aligns existing pre-trained models rather than requiring pretraining from scratch. Analyzing multilingual BERT. present a series of probing experiments to better understand multilingual BERT, and they find that transfer is possible even between dissimilar languages, but that it works better between languages that are typologically similar. They conclude that BERT is remarkably multilingual but falls short for certain language pairs. We first briefly describe multilingual BERT . Like monolingual BERT, multilingual BERT is pre-trained on sentences from Wikipedia to perform two tasks: masked word prediction, where it must predict words that are masked within a sentence, and next sentence prediction, where it must predict whether the second sentence follows the first one. The model is trained on 104 languages, with each batch containing training sentences from each language, and it uses a shared vocabulary formed by WordPiece on the 104 Wikipedias concatenated . In the following sections, we describe how to define, evaluate, and improve contextual alignment. Given two languages, a model is in contextual alignment if it has similar representations for word pairs within parallel sentences. More precisely, suppose we have N parallel sentences is a source-target sentence pair. Also, let each sentence pair (s, t) have word pairs, denoted a(s, t) = {(i 1, j 1),..., (i m, j m)}, containing position tuples (i, j) such that the words s i and t j are translations of each other. 1 We will use f to represent a pre-trained model such that f (i, s) is the contextual embedding for the ith word in s. Published as a conference paper at ICLR 2020 As an example, we might have the following sentence pair: Then, using the parallel corpus C, we can measure the contextual alignment of the model f using its accuracy in contextual word retrieval. In this task, the model is presented with two parallel corpora, and given a word within a sentence in one corpus, it must find the correct word and sentence in the other. Specifically, we can define a nearest neighbor retrieval function where i and j denote the position within a sentence and sim is a similarity function. The accuracy is then given by the percentage of exact matches over the entire corpus, or where I represents the indicator function. We can perform the same procedure in the other direction, where we pull target words given source words, so we report the average between the two directions. As our similarity function, we use CSLS, a modified version of cosine similarity that mitigates the hubness problem, with neighborhood size 10 (a). One additional point is that this procedure can be made more or less contextual based on the corpus: a corpus with more occurrences for each word type requires better representations of context. Therefore, we also test non-contextual word retrieval by removing all but the first occurrence of each word type. Given parallel data, these word pairs can be procured in an unsupervised fashion using standard techniques developed by the machine translation community . While these methods can be noisy, by running the algorithm in both the source-target and target-source directions and only keeping word pairs in their intersection, we can trade-off coverage for accuracy, producing a reasonably high-precision dataset . To improve the alignment of the model f with respect to the corpus C, we can encapsulate alignment in the loss function where we sum the similarities between word pairs. Because the CSLS metric is not easily optimized, we instead use the squared error loss, or sim(2 . However, note that this loss function does not account for the informativity of f ; for example, it is zero if f is constant. Therefore, at a high level, we would like to minimize L(f ; C) while maintaining some aspect of f that makes it useful, e.g. its high accuracy when fine-tuned on downstream tasks. Letting f 0 denote the initial pre-trained model before alignment, we achieve this goal by defining a regularization term which imposes a penalty if the target language embeddings stray from their initialization. Then, we sample minibatches B ⊂ C and take gradient steps of the function L(f ; B) + λR(f ; B) directly on the weights of f, which moves the source embeddings toward the target embeddings while preventing the latter from drifting too far. In our experiments, we set λ = 1. In the multilingual case, suppose we have k parallel corpora C 1,..., C k, where each corpus has a different source language with the target language as English. Then, we sample equal-sized batches B i ⊂ C i from each corpus and take gradient steps on, which moves all of the non-English embeddings toward English. Note that this alignment method departs from prior work, in which each non-English language is rotated to match the English embedding space through individual learned matrices. Specifically, the most widely used post-hoc alignment method learns a rotation W applied to the source vectors to minimize the distance between parallel word pairs, or This problem is known as the Procrustes problem and can be solved in closed form . This approach has the nice property that the vectors are only rotated, preserving distances and therefore the semantic information captured by the embeddings . However, rotation requires the strong assumption that the embedding spaces are roughly isometric (Søgaard et al., 2018), an assumption that may not hold for contextual pre-trained models because they represent more aspects of a word than just its type, i.e. context and syntax, which are less likely to be isomorphic between languages. Given that past work has also found independent alignment per language pair to be inferior to joint training , another advantage of our method is that the alignment for all languages is done simultaneously. As our dataset, we use the Europarl corpora for English paired with Bulgarian, German, Greek, Spanish, and French, the languages represented in both Europarl and XNLI . After tokenization , we produce word pairs using fastAlign and keep the one-to-one pairs in the intersection . We use the most recent 1024 sentences as the test set, the previous 1024 sentences as the development set, and the following 250K sentences as the training set. Furthermore, we modify the test set accuracy calculation to only include word pairs not seen in the training set. We also remove any exact matches, e.g. punctuation and numbers, because BERT is already aligned for these pairs due to its shared vocabulary. Given that parallel data may be limited for low-resource language pairs, we also report numbers for 10K and 50K parallel sentences. Given that there has been a long line of work on word vector alignment (; a; , inter alia), we also compare BERT to a sentence-augmented fastText baseline . , we first normalize, then mean-center, then normalize the word vectors, and we then learn a rotation with the same parallel data as in the contextual case, as described in Equation 1. We also strengthen this baseline by including sentence information: specifically, during word retrieval, we concatenate each word vector with a vector representing its sentence. Following Rücklé et al., we compute the sentence vector by concatenating the average, maximum, and minimum vector over all of the words in the sentence, a method that was shown to be state-of-the-art for a suite of cross-lingual tasks. We also experimented with other methods, such as first retrieving the sentence and then the word, but found this method ed in the highest accuracy. As a , the fastText vectors are 1200-dimensional, while the BERT vectors are 768-dimensional. The next step is to determine whether better alignment improves cross-lingual transfer. As our downstream task, we use the XNLI dataset, where the English MultiNLI development and test sets are human-translated into multiple languages (b;). Given a pair of sentences, the task is to predict whether the first sentence implies the second, where there are three labels: entailment, neutral, or contradiction. Starting from either the base or aligned multilingual BERT, we train on English and evaluate on Bulgarian, German, Greek, Spanish, and French, the XNLI languages represented in Europarl. As our architecture, following , we apply a linear layer followed by softmax on the [CLS] embedding of the sentence pair, producing scores for each of the three labels. The model is trained using cross-entropy loss and selected based on its development set accuracy averaged across all of the languages. As a fully-supervised ceiling, we also compare to models trained and tested on the same language, where for the non-English training data, we use the machine translations of the English MultiNLI training data provided by Conneau et al. (2018b). While the quality of the training data is affected by the quality of the MT system, this comparison nevertheless serves as a good approximation for the fully-supervised setting. and two rotation-based methods, sentence alignment and word alignment . We also include the current state-of-the-art zero-shot achieved by XLM . Rotation-based methods provide small gains on some languages but not others. On the other hand, after fine-tuning-based alignment, Bulgarian and Greek match the translate-train ceiling, while German, Spanish, and French close roughly one-third of the gap. Table 2: Zero-shot accuracy on the XNLI test set, where we align BERT with varying amounts of parallel data. The method scales with the amount of data but achieves a large fraction of the gains with 50K sentences per language pair. First, we test whether alignment improves multilingual BERT by applying the models to zero-shot XNLI, as displayed in Table 1. We see that our alignment procedure greatly improves accuracies, with all languages seeing a gain of at least 1%. In particular, the Bulgarian and Greek zero-shot numbers are boosted by almost 5% each and match the translate-train numbers, suggesting that the alignment procedure is especially effective for languages that are initially difficult for BERT. We also run alignment for more distant language pairs (Chinese, Arabic, Urdu) and find similar , which we report in the appendix. Comparing to rotation-based methods , we find that a rotation produces small gains for some languages, namely Bulgarian, German, and Spanish, but is sub-optimal overall, providing evidence that the increased expressivity of our proposed procedure is beneficial for contextual alignment. We explore this comparison more in Section 5.1. Given that our goal is zero-shot transfer, we cannot expect to always have large amounts of parallel data. Therefore, we also characterize the performance of our alignment method with varying amounts of data, as displayed in Table 2. We find that it improves transfer with as little as 10K sentences per language, making it a promising approach for low-resource languages. Table 3: Word retrieval accuracy for the aligned sentence-augmented fastText baseline and BERT pre-and post-alignment. Across languages, base BERT has variable accuracy while fine-tuningaligned BERT is consistently effective. Fine-tuned BERT also matches fastText in a version of the task where context is not necessary, suggesting that our method matches the type-level alignment of fastText while also aligning context. In the following sections, we present word retrieval to both compare our method to past work and better understand the strengths and weaknesses of multilingual BERT. Table 3 displays the word retrieval accuracies for the aligned sentence-augmented fastText baseline and BERT pre-and postalignment. First, we find that in contextual retrieval, fine-tuned BERT outperforms fastText, which outperforms rotation-aligned BERT. This supports the intuition that aligning large pre-trained models is more difficult than aligning word vectors, given that a rotation, at least when applied naively, produces sub-par alignments. In addition, fine-tuned BERT matches the performance of fastText in non-contextual retrieval, suggesting that our alignment procedure overcomes these challenges and achieves type-level alignment that matches non-contextual approaches. In the appendix, we also provide examples of aligned BERT disambiguating between different meanings of a word, giving qualitative evidence of the benefit of context alignment. We also find that before alignment, BERT's performance varies greatly between languages, while after alignment it is consistently effective. In particular, Bulgarian and Greek initially have very low accuracies. This phenomenon is also reflected in the XNLI numbers (Table 1), where Bulgarian and Greek receive the largest boosts from alignment. Examining the connection between alignment and zero-shot more closely, we find that the word retrieval accuracies are highly correlated with downstream zero-shot performance (Figure 2), supporting our evaluation measure as predictive of cross-lingual transfer. The language discrepancies are also consistent with a hypothesis by to explain BERT's multilingualism. They posit that due to the shared vocabulary, shared words between languages, e.g. numbers and names, are forced to have the same representation. Then, due to the masked word prediction task, other words that co-occur with these shared words also receive similar representations. If this hypothesis is true, then languages with higher lexical overlap with English are likely to experience higher transfer. As an extreme form of this phenomenon, Bulgarian and Greek have completely different scripts and should experience worse transfer than the common-script languages, an intuition that is confirmed by the word retrieval and XNLI accuracies. The fact that all languages are equally aligned with English post-alignment suggests that the pre-training procedure is suboptimal for these languages. Table 4: Accuracy by part-of-speech tag for non-contextual word retrieval. To achieve better word type coverage, we do not remove word pairs seen in the training set. The tags are grouped into lexically overlapping, closed-class, and open-class groups. The "Particle," "Symbol," "Interjection," and "Other" tags are omitted. Fraction correct Aligned BERT Base BERT Figure 3: Contextual word retrieval accuracy plotted against difference in frequency rank between source and target. The accuracy of base BERT plummets for larger differences, suggesting that its alignment depends on word pairs having similar usage statistics. Next, to gain insight into the multilingual pre-training procedure, we analyze the accuracy broken down by part-of-speech using the Universal Part-of-Speech Tagset , annotated using polyglot for Bulgarian and spaCy for the other languages, as displayed in Table 4. Unsurprisingly, multilingual BERT has high alignment out-of-the-box for groups with high lexical overlap, e.g. numerals, punctuation, and proper nouns, due to its shared vocabulary. We further divide the remaining tags into closed-class and open-class, where closed-class parts-of-speech correspond to fixed sets of words serving grammatical functions (e.g. determiner, preposition, conjunction, pronoun, and auxiliary), while open-class parts-of-speech correspond to lexical words (e.g. noun, adverb, adjective, verb). Interestingly, we see that base BERT has consistently lower accuracy for closed-class versus open-class categories (0.70 vs 0.54), but that this discrepancy disappears after alignment (0.88 vs 0.89). From this closed-class vs open-class difference, we hypothesize that BERT's alignment of a particular word pair is influenced by the similarity of their usage statistics. Specifically, given that BERT is trained through masked word prediction, its embeddings are in large part determined by the co-occurrences between words. Therefore, two words that are used in similar contexts should be better aligned. This hypothesis provides an explanation of the closed-class vs open-class difference: closed-class words are typically grammatical, so they are used in similar ways across typologically similar languages. Furthermore, these words cannot be substituted for one another due to their grammatical function. Therefore, their usage statistics are a strong signature that can be used for alignment. On the other hand, open-class words can be substituted for one another: for example, in most sentences, the noun tokens could be replaced by a wide range of semantically dissimilar nouns with the sentence remaining syntactically well-formed. By this effect, many nouns have similar co-occurrences, making them difficult to align through masked word prediction alone. To further test this hypothesis, we plot the word retrieval accuracy versus the difference between the frequency rank of the target and source word, where this difference measures discrepancies in usage, as depicted in Figure 3. We see that accuracy drops off significantly as the source-target difference increases, supporting our hypothesis. Furthermore, this shortcoming is remedied by alignment, revealing another systematic deficiency of multilingual pre-training. Given that the degree of alignment is causally predictive of downstream cross-lingual transfer, contextual alignment proves to be a useful concept for understanding and improving multilingual pretrained models. Given small amounts of parallel data, our alignment procedure improves multilingual BERT and corrects many of its systematic deficiencies. Contextual word retrieval also provides useful new insights into the pre-training procedure, opening up new avenues for analysis. Table 5: Zero-shot accuracy on the XNLI test set with more languages, where we use 20K parallel sentences for each language paired with English. This confirms that the alignment method works for distant languages and a variety of parallel corpora, including Europarl, MultiUN, and Tanzil, which contains sentences from the Quran (; ;). A APPENDIX For both alignment and XNLI optimization, we use a learning rate of 5 × 10 −5 with Adam hyperparameters β = (0.9, 0.98), = 10 −9 and linear learning rate warmup for the first 10% of the training data. For alignment, the model is trained for one epoch, with each batch containing 2 sentence pairs per language. For XNLI, each model is trained for 3 epochs with 32 examples per batch, and 10% dropout is applied to the BERT embeddings. In Table 5, we report numbers for additional languages, where we align a single BERT model for all eight languages and then fine-tune on XNLI. We use 20K sentences per language, where we use the MultiUN corpus for Arabic and Chinese , the Tanzil corpus for Urdu , and the Europarl corpus for the other five languages . This confirms that the alignment method works for a variety of languages and corpora. Furthermore, the Tanzil corpus consists of sentences from the Quran, suggesting that the method works even when the parallel corpus and downstream task contain sentences from entirely different domains. In this section, we qualitatively show that aligned BERT is able to disambiguate between different occurences of a word. First, we find two meanings of the word "like" occurring in the EnglishGerman Europarl test set. Note also that in the second and third example, the two senses of "like" occur in the same sentence. • This empire did not look for colonies far from home or overseas, like most Western European States, but close by. Dieses Reich suchte seine Kolonien nicht weit von zu Hause und in bersee wie die meisten westeuropäischen Staaten, sondern in der unmittelbaren Umgebung. • Like other speakers, I would like to support the call for the arms embargo to remain. Wie andere Sprecher, so möchte auch ich den Aufruf zur Aufrechterhaltung des Waffenembargos untersttzen. • Like other speakers, I would like to support the call for the arms embargo to remain. Wie andere Sprecher, so möchte auch ich den Aufruf zur Aufrechterhaltung des Waffenembargos untersttzen. • I would also like, although they are absent, to mention the Commission and the Council. Ich möchte mir sogar erlauben, die Kommission und den Rat zu nennen, auch wenn sie nicht anwesend sind. Multiple meanings of "order": • Moreover, the national political elite had to make a detour in Ambon in order to reach the civil governor's residence by warship. In Ambon mußte die politische Spitze des Landes auch noch einen Umweg machen, um mit einem Kriegsschiff die Residenz des Provinzgouverneurs zu erreichen. • Although the European Union has an interest in being surrounded by large, stable regions, the tools it has available in order to achieve this are still very limited. Der Europäischen Union ist zwar an großen stabilen Regionen in ihrer Umgebung gelegen, aber sie verfgt nach wie vor nur ber recht begrenzte Instrumente, um das zu erreichen. • We could reasonably expect the new Indonesian government to take action in three fundamental areas: restoring public order, prosecuting and punishing those who have blood on their hands and entering into a political dialogue with the opposition. Von der neuen indonesischen Regierung darf man mit Fug und Recht drei elementare Maß-nahmen erwarten: die Wiederherstellung deröffentlichen Ordnung, die Verfolgung und Bestrafung derjenigen, an deren Händen Blut klebt, und die Aufnahme des politischen Dialogs mit den Gegnern. • Firstly, I might mention the fact that the army needs to be reformed, secondly that a stable system of law and order needs to be introduced. Ich nenne hier an erster Stelle die notwendige Reform der Armee, ferner die Einfhrung eines stabilen Systems rechtsstaatlicher Ordnung. • Financial support is needed to enable poor countries to take part in these court activities. Arme Länder müssen finanziell unterstützt werden, damit auch sie sich an der Arbeit des Gerichtshofs beteiligen können. • We must help them and ensure that a proper action plan is implemented to support their work. Es gilt einen wirklichen Aktionsplan auf den Weg zu bringen, um die Arbeit dieser Organisationen zu unterstützen. • So I hope that you will all support this resolution condemning the abominable conditions of prisoners and civilians in Djibouti. Ich hoffe daher, daß Sie alle diese Entschließung befürworten, die die entsetzlichen Bedingungen von Inhaftierten und Zivilpersonen in Dschibuti verurteilt. • It would be difficult to support a subsidy scheme that channelled most of the aid to the large farms in the best agricultural regions. Es wäre auch problematisch, ein Beihilfesystem zu befürworten, das die meisten Beihilfen in die großen Betriebe in den besten landwirtschaftlichen Gebieten lenkt. Multiple meanings of "close": • This empire did not look for colonies far from home or overseas, like most Western European States, but close by. Dieses Reich suchte seine Kolonien nicht weit von zu Hause und in bersee wie die meisten westeuropäischen Staaten, sondern in der unmittelbaren Umgebung. • In addition, if we are to shut down or refuse investment from every company which may have an association with the arms industry, then we would have to close virtually every American and Japanese software company on the island of Ireland with catastrophic consequences. Wenn wir zudem jedes Unternehmen, das auf irgendeine Weise mit der Rstungsindustrie verbunden ist, schließen oder Investitionen dieser Unternehmen unterbinden, dann mßten wir so ziemlich alle amerikanischen und japanischen Softwareunternehmen auf der irischen Insel schließen, was katastrophale Auswirkungen hätte. • On the other hand, the deployment of resources left over in the Structural Funds from the programme planning period 1994 to 1999 is hardly worth considering as the available funds

We propose procedures for evaluating and strengthening contextual embedding alignment and show that they both improve multilingual BERT's zero-shot XNLI transfer and provide useful insights into the model.

Represent the short summary for finding the corresponding scientific article (long).

Meta-learning methods used for vision, directly applied to NLP, perform worse than nearest neighbors on new classes; we can do better with distributional signatures.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Neural networks have reached outstanding performance for solving various ill-posed inverse problems in imaging. However, drawbacks of end-to-end learning approaches in comparison to classical variational methods are the requirement of expensive retraining for even slightly different problem statements and the lack of provable error bounds during inference. Recent works tackled the first problem by using networks trained for Gaussian image denoising as generic plug-and-play regularizers in energy minimization algorithms. Even though this obtains state-of-the-art on many tasks, heavy restrictions on the network architecture have to be made if provable convergence of the underlying fixed point iteration is a requirement. More recent work has proposed to train networks to output descent directions with respect to a given energy function with a provable guarantee of convergence to a minimizer of that energy. However, each problem and energy requires the training of a separate network. In this paper we consider the combination of both approaches by projecting the outputs of a plug-and-play denoising network onto the cone of descent directions to a given energy. This way, a single pre-trained network can be used for a wide variety of reconstruction tasks. Our show improvements compared to classical energy minimization methods while still having provable convergence guarantees. In many image processing tasks an observed image f is modeled as the of the transformation of a clean imageû under a known (linear) operator A and unknown noise ξ, f = Aû + ξ. (In most cases, the problem of reconstructingû from f and A is ill-posed and can thus not be solved by a simple inversion of A, giving rise to the field of regularization theory with iterative or variational methods, see e.g. for an overview. In recent years neural networks were very successful in learning a direct mapping G(f) ≈û for a variety of problems such as deblurring, denoising, super-resolution, demosaicing and MRI-or CT-reconstruction. Even though this works well in practice, there are rarely any guarantees on the behaviour of neural networks on unseen data, making them difficult to use in safety-critical applications. Moreover, for each problem and type of noise a separate network has to be trained. In contrast, classical variational methods try to find the solution by the minimization of a suitable energy function of the formû where H f is a data fidelity term, for example commonly chosen as H f (u) = 1 2 ||Au − f || 2, and R is a regularization function that models prior knowledge about the solution, e.g. the popular total variation (TV) regularization, R(u) = ∇u 1,. While minimizers of come with many desirable theoretical guarantees, regularizations like the TV often cannot perfectly capture the complex structure of the space of natural images. To combine the advantages of powerful feed-forward networks and model-based approaches like, authors have considered various hybrid models like learning regularizers (e.g. ), designing networks architectures that resemble the structure of minimization algorithms or differential equations, e.g., interleaving networks with classical optimization steps, or using the parametrization of networks as a regularization for, see e.g.. A particularly flexible approach arises from, where proximal operators with respect to the regularizer are replaced by arbitrary denoising operators, with recent works focusing on the use of denoising networks. While such approaches allow to tackle different inverse problems with the same neural network, the derivation of theoretical guarantees -even in terms of the convergence of the ing algorithmic scheme -remains difficult, see or some discussion in, unless the denoiser satisfies particular properties. The starting point of the above-mentioned algorithmic schemes that utilize denoising networks to regularize model-based inverse problems are methods for the minimization of. While most works focus on primal-dual / ADMM approaches, their convergence analysis is quite delicate even in a setting in which one still minimizes (nonconvex) energies, such that we turn to two simpler methods, gradient descent and proximal gradient methods, where Following the idea of, considering either a gradient descent or a proximal step on the regularization as a generic denoising operation gives rise to the following two algorithmic schemes, where G denotes any kind of denoiser, e.g. a convolutional neural network, and we define ρ(u for the sake of brevity of notation. We refer to for a more detailed derivation. Algorithmic schemes like or combine the model-based flexibility of energy minimization methods (i.e. explicit modelling of H f) with the expressive power of deep neural networks G. Unfortunately -despite their success in various practical applications -schemes like or remain dangerous to be used: Figure 1 shows the of running the iteration with H f = 0 on a noisy input image f = u 0 for 100 and 800 iterations using a DnCNN preimplemented in Matlab as the denoiser G. As we can see the image gets completely distorted. Even more strikingly, the range of the image increased from values in to an interval of [−185, 218] within the first 1000 iterations. Clearly, the algorithmic scheme diverges. A natural condition for the provable convergence of a scheme like (at least along subsequences) would be a 1-Lipschitz continuous operator G. There has been previous work on computing upper bounds for the best Lipschitz constant of a network and using it to enforce a user defined Lipschitz constant L during training time but we found that enforcing non-expansiveness drastically decreased the denoising performance. The problem of computing the best Lipschitz constant, in hope of improving those , was recently proved to be NP-hard and thus is infeasible. Therefore, we adapt the recent idea proposed in to safeguard neural networks by forcing them to predict a descent direction to a given model-based energy, such that it can be used within a line search algorithm to guarantee convergence. More precisely, at any given estimate u and model-based energy E the authors use the Euclidean projection onto the half space as the last layer of their network. Even though the ing algorithm converges to the minimizer of E, experiments showed significantly higher peaks of the PSNR value in early iterations compared to classical gradient descent on E. Intuitively, the descent direction proposed by the network pushes the iteration closer towards the distribution of the training data than a usual gradient descent step. While the approach of has to train a separate network for each inverse problem and each type of noise, we investigate the combination of the flexible algorithmic schemes and with the idea from to project onto the half-space of descent directions to safeguard the underlying algorithm. In the following G will always refer to a generic denoising network, like DnCNN. We assume that E(u) = H f (u) + R(u) is a continuously differentiable, strictly convex and coercive energy function. As a first step, we simply rewrite the algorithmic schemes and in such a way that they resemble a gradient descent iteration, i.e., such that we can interpret as "update directions" of the respective algorithmic schemes. Because the plain iterations and can easily be divergent, we safeguard them by projecting them onto the half-space of descent directions C(γ, ∇E(u k)), i.e., Note that we replaced the averaging of the gradient descent and denoising step in by an abitrary convex combination using a parameter α to determine the respective influence of the data term and the denoising more flexibly. After computing the above directions d k, we update our iterates using with a step size t k chosen based on a backtracking line-search mechanism similar to. Under weak additional conditions, the latter guarantees the convergence of the proposed scheme to the minimizer of E. Such a minimizer could of course be determined by any classical algorithm, but we hope for to yield a better path towards the true minimizer, and consider a discrepancy principle for stopping the iteration before convergence. More precisely, we terminate as soon as for H f (û) being an estimate on the (data-term-dependent measure of the) noise level of the considered problem, and β being a scaling factor (typically close to 1). We tested our implementation with the image reconstruction tasks of Gaussian deblurring with standard deviation 1 and 4× single image super resolution. In both cases we added Gaussian noise with standard deviation 0.02 to the corrupted image. We chose the PyTorch implementation 1 of DnCNN pre-trained on a noise level of 0.1 as our denoising network. Our surrogate energy uses a TV regularization with Huber-norm instead of the 1 -norm. As our data term we choose H f (u) = operator. The best hyperparameters for all methods were found with a grid search. In all experiments, for scheme (conv), α = 0 was the best choice for any τ, indicating that the gradient descent step on the data term does not yield much additional information, assumably because the projection onto C(γ, ∇E(u)) which depends on the gradient of the data term anyway. When using (prox), we empirically found τ = 30 to be the best choice. For the projection onto the half-space of descent directions, we we used γ = 5 for both methods for deblurring, and γ = 50 in (conv) and γ = 1 in (prox) for super resolution. For fairness, the classical gradient descent was also implemented using backtracking line search. Figure 2 shows the reconstruction quality of the current iterate compared to ground truth over a span of 500 iterations. The PSNR quickly peaks before slowly converging to the fixed point of the surrogate energy which is consistent with the of. Notably, the convex combination method peaks earlier but not as high as the prox method. Tables 1 and 2 show using early stopping using a discrepancy principle. On all test images our prox scheme beats gradient descent. Table 1: PSNR values for deblurring for varying images and stopping criteria. The algorithm was stopped when H f (u k) < βH f (û). best refers to the highest PSNR over 500 iterations and a "*" means that the stopping criterion was not triggered such that the last iteration was used instead. Table 2: PSNR values for super resolution for varying images and stopping criteria. The algorithm was stopped when H f (u k) < βH f (û). best refers to the highest PSNR over 500 iterations. We combine deep learning and energy minimization methods for solving inverse problems in image reconstruction into a provably convergent algorithmic scheme. Still, our approach is able to generalize to different problems with a single denoising network and without the need to retrain if that problem changes. We were able to reach better than the energy minimization baseline in our experiments, and are happy to elaborate on the above aspects in the NeurIPS workshop.

Represent the TLDR for finding the corresponding scientific article

We use neural networks trained for image denoising as plug-and-play priors in energy minimization algorithms for image reconstruction problems with provable convergence.

represent the tldr for finding the corresponding scientific article.

Example:
Provided: "Improving label efficiency through multi-task learning on auditory data" Match: "Collecting high-quality, large scale datasets typically requires significant resources. The aim of the present work is to improve the label efficiency of large neural networks operating on audio data through multitask learning with self-supervised tasks on unlabeled data. To this end, we trained an end-to-end audio feature extractor based on WaveNet that feeds into simple, yet versatile task-specific neural networks. We describe three self-supervised learning tasks that can operate on any large, unlabeled audio corpus. We demonstrate that, in a scenario with limited labeled training data, one can significantly improve the performance of a supervised classification task by simultaneously training it with these additional self-supervised tasks. We show that one can improve performance on a diverse sound events classification task by nearly 6\% when jointly trained with up to three distinct self-supervised tasks. This improvement scales with the number of additional auxiliary tasks as well as the amount of unsupervised data. We also show that incorporating data augmentation into our multitask setting leads to even further gains in performance. Although audio tag classification does not require the fine temporal resolution found in raw audio 91 waveforms, our chosen auxiliary tasks (or any arbitrary auditory task for which we may desire our 92 model to be sufficient) require higher temporal resolutions. To satisfy this, we chose to build our 93 model following the WaveNet architecture. are processed using small, task-specific neural networks built atop a task-agnostic trunk. The trunk architecture principally follows the structure of WaveNet, with several blocks of stacked, dilated, and causal convolutions between every convolution layer. Outputs from the trunk are fed into task-specific heads (details in Section 3.1).As shown Figure 1, our WaveNet trunk is composed of N blocks, where each block consists of S dilated causal convolution layers, with dilation factors increasing from 1 to 2 S − 1, residual connections and saturating nonlinearities. We label the blocks using b = 1, · · ·, N. We use indices ∈ [1 + (b − 1)S, bS] to label layers in block b. Each layer,, of the WaveNet trunk consists of a "residual atom" which involves two computations, labeled as "Filter" and "Gate" in the figure. Each residual atom computation produces a hidden state vector h and a layer output x defined via DISPLAYFORM0 where denotes element-wise products, represents the regular convolution operation, denotes dilated convolutions with a dilation factor of 2 mod bS if is a layer in block b + 1, σ denotes the sigmoid function and W gate and W f ilter are the weights for the gate and filter, respectively. The first (= 0) layer -represented as the initial stage marked "1 × 1 Conv" in Figure 1 -applies causal convolutions to the raw audio waveforms X = (X 1, X 2, · · ·, X T), sampled at 16 kHz, to produce an output DISPLAYFORM1 Given the structure of the trunk laid out above, any given block b has an effective receptive field of 1 + b(2 S − 1). Thus the total effective receptive field of our trunk is τ = 1 + N (2 S − 1). Following an extensive hyperpameter search over various configurations, we settled on [N = 3] blocks comprised of [S = 6] layers each for our experiments. Thus our trunk has a total receptive field of τ = 190, which corresponds to about 12 milliseconds of audio sampled at 16kHz. As indicated above, each task-specific head is a simple neural network whose input data is first 102 constrained to pass through a trunk that it shares with other tasks. Each head is free to process this 103 input to its advantage, independent of the other heads. Each task also specifies its own objective function, as well as a task-specific optimizer, with cus-105 tomized learning rates and annealing schedules, if necessary. We arbitrarily designate supervised 106 tasks as the primary tasks and refer to any self-supervised tasks as auxiliary tasks. In the experiments 107 reported below, we used "audio tagging" as the primary supervised classification task and "next-step 108 prediction", "noise reduction" and "upsampling" as auxiliary tasks training on various amounts of unlabeled data. The parameters used for each of the task specific heads can be found in TAB1 accompanying supplement to this paper. Figure 2: The head architectures were designed to be simple, using only as few layers as necessary to solve the task. Simpler head architectures force the shared trunk to learn a representation suitable for multiple audio tasks. The next-step prediction task can be succinctly formalized as follows: given a sequence 113 {x t−τ +1, · · ·, x t} of frames of an audio waveform, predict the next value x t+1 in the sequence. This prescription allows one to cheaply obtain arbitrarily large training datasets from an essentially 115 unlimited pool of unlabeled audio data. Our next-step prediction head is a 2-layer stack of 1×1 convolutional layers with ReLU nonlinearities 117 in all but the last layer. The first layer contains 128 units, while the second contains a single output unit. The head takes in τ frames of data from the trunk, where τ is the trunk's effective receptive field, and 119 produces an output which represents the model's prediction for the next frame of audio in the sequence. The next-step head treats this as a regression problem, using the mean squared error of the difference 121 between predicted values and actual values as a loss function, i.e. given inputs {x t−τ +1, · · ·, x t}, the head produces an output y t from which we compute a loss L MSE (t) = (y t − x t+1) 2 and then 123 aggregate over the frames to get the total loss. We would like to note that the original WaveNet implementation treated next-step prediction as a 125 classification problem, instead predicting the bin-index of the audio following a µ-law transform. We 126 found that treating the task as a regression problem worked better in multitask situations but make no claims on the universality of this choice. In defining the noise reduction task, we adopt the common approach of treating noise as an additive 130 random process on top of the true signal: if {x t} denotes the clean raw audio waveform, we obtain 131 the noisy version viax t:= x t + ξ t where ξ t an arbitrary noise process. For the denoising task, the 132 model is trained to predict the clean sample, x t, given a window x t− well-adapted to solving either task. Thus, our noise reduction head has a structure similar to the 136 next-step head. It is trained to minimize a smoothed L1 loss between the clean and noisy versions of 137 the waveform inputs, i.e. for each frame t, the head produces an outputŷ t, and we compute the loss DISPLAYFORM0 and then aggregate over frames to obtain the total loss. We used the smooth L1 loss because it 139 provided a more stable convergence for the denoising task than mean squared error. In the same spirit as the denoising task, one can easily create an unsupervised upsampling task Again, given the formal similarity of the upsampling task to the next-step prediction and noise-151 reduction tasks, we used an upsampling head with a structure virtually identical to those described 152 above. As with the denoising task, we used a smooth L1 loss function (see eqn. FORMULA2 having manually-verified labels and the remaining 5763 having non-verified labels, meaning they 177 were automatically categorized using user-provided metadata. The test set is composed of 1600 audio 178 clips with manually-verified labels which are used for the final scoring. The Librispeech dataset 1 (comprised of read English speech sampled at 16 kHz) was used as a proxy for a large unlabeled dataset. The models described below were trained using clips from either the 182 "train-clean-100" or "train-other-500 versions". Models trained with 5, 50 and 100 hours of unlabeled 183 data were sourced from "train-clean-100", while the model trained with 500 hours was sourced entirely from "train-other-500". Due to memory constraints, we limited the duration of each utterance 185 to 2 seconds which we obtained by cropping from a random position in the original clip. This dataset 186 was only used to train the auxiliary tasks. We trained the model using raw audio waveform inputs taken from the FSDKaggle2018 and Lib-189 rispeech datasets. All code for the experiments described here was written in the PyTorch framework. All audio samples were first cropped to two seconds in duration and downsampled to 16 kHz. To normalize for the variation in onset times for different utterances, the 2 seconds were randomly both the main task and the auxiliary tasks, heuristically favoring performance on the main task. We jointly trained the model on all tasks simultaneously by performing a forward pass for each task, important parameters of the model can be found in TAB4 of the accompanying supplement to this 217 paper. As discussed above, we used audio tagging as the main task to investigate whether supervised First, we trained a purely supervised model on 2 seconds of non-silence audio extracted using random In this experiment, we added each of the self-supervised tasks to the baseline model discussed above, simultaneously training them using 100 hours of unlabeled data sampled from the Librispeech dataset 236 along with the main supervised task. We notice that, addition of any self-supervised task showed an average improvement of 4.6% to the MAP@3 score compared to the main task's baseline performance. Adding a pair of tasks gave an average improvement of 4.55% over baseline, showing no improvement over adding a single task. Training with three additional tasks yielded the best with an improvement of 5.33% over the main task. Looking at MAP@3 scores throughout training showed that convergence in every multitask setting was stable, with gradual improvements for increasing number of tasks. The best performance values on the test sets for a sequence of task additions can be found in TAB1.The set of experiments described above demonstrate that, for a fixed amount of unlabeled data (100 hours), simultaneously training a supervised task with various self-supervised tasks yields a significant improvement in the main task's performance. To further test how performance changes with increasing amounts of data, we re-trained our model while varying the amount of unlabeled data used to train the auxiliary tasks. We noticed that even without any additional unlabeled data, the MAP@3 score with three additional tasks was significantly better than the score obtained on a single task. This demonstrates that addition of self-supervised tasks improves the performance of main task. Increasing the size of the unlabeled data for the auxiliary tasks increases the size of the multitask benefit (Figure 3).The MAP@3 Scores at different levels of unlabaled data showed progressive improvement to 0.656, 0.669, with 5 and 50 hours respectively. We observed a peak MAP@3 score of 0.694 with 500 hours of unlabeled data, which is an improvement of 8.94% over the main task's baseline performance. Next, we explore several approaches to data augmentation and compare them with multitask learning. Previous work has demonstrated the effectiveness of data augmentation through simple techniques, such as noise injection, and pitch shifting. We compared our proposed method with traditional data augmentation strategies by retraining our model only for the main task after applying the aforementioned augmentations to the FSDKaggle2018 training data. The MAP@3 values for the data augmentation experiments on the test sets can be found in TAB2. We observed a peak MAP@3 score of 0.703 with pitch shifting augmentation which is similar in scale to that of our best multitask performance gains. In an attempt to observe how both the techniques work together, we combined data augmentation with multitask learning and obtain an MAP@3 score of 0.726 which was the best score among all the experiments we conducted. Figure 3: Improved MAP@3 scores with increasing amounts of unlabeled data. Shown are the MAP@3 scores on test set when the main task is trained with 3 auxiliary tasks with 0, 5, 50, 100, and 500 hours of unlabeled data respectively. The amount of labelled data is held constant for the whole experiment. We see a smooth increase in performance with increasing amounts of unlabeled data. where one has a limited quantity of labeled data. We have also shown that the performance of the 244 supervised task improves by increasing either the number of auxiliary self-supervised tasks or the 245 quantity of unlabeled data or both. We attain a peak performance boost of 8.94% over the baseline 246 with the inclusion of 3 self-supervised tasks when trained with additional 500 hours of unlabeled data. Finally, our multitask learning scheme further benefits when the training data for the data-constrained 248 task is augmented using standard techniques. Since our suggest that the performance gain with 249 our approach is additive when used with data augmentation, it may be interesting to use multitask 250 learning with other augmentation approaches to observe if they complement each other in different 251 settings. We have strived to systematically present our within a coherent multitask learning framework."

For complex constraints in which it is not easy to estimate the gradient, we use the discounted penalty as a guiding signal. We prove that under certain assumptions it converges to a feasible solution.