Question: Represent the article for finding a single-sentence TLDR

Question: Modeling hypernymy, such as poodle is-a dog, is an important generalization aid to many NLP tasks, such as entailment, relation extraction, and question answering. Supervised learning from labeled hypernym sources, such as WordNet, limit the coverage of these models, which can be addressed by learning hypernyms from unlabeled text. Existing unsupervised methods either do not scale to large vocabularies or yield unacceptably poor accuracy. This paper introduces {\it distributional inclusion vector embedding (DIVE)}, a simple-to-implement unsupervised method of hypernym discovery via per-word non-negative vector embeddings which preserve the inclusion property of word contexts. In experimental evaluations more comprehensive than any previous literature of which we are aware---evaluating on 11 datasets using multiple existing as well as newly proposed scoring functions---we find that our method provides up to double the precision of previous unsupervised methods, and the highest average performance, using a much more compact word representation, and yielding many new state-of-the-art . In addition, the meaning of each dimension in DIVE is interpretable, which leads to a novel approach on word sense disambiguation as another promising application of DIVE. Numerous applications benefit from compactly representing context distributions, which assign meaning to objects under the rubric of distributional semantics. In natural language processing, distributional semantics has long been used to assign meanings to words (that is, to lexemes in the dictionary, not individual instances of word tokens). The meaning of a word in the distributional sense is often taken to be the set of textual contexts (nearby tokens) in which that word appears, represented as a large sparse bag of words (SBOW). Without any supervision, word2vec BID22, among other approaches based on matrix factorization BID20, successfully compress the SBOW into a much lower dimensional embedding space, increasing the scalability and applicability of the embeddings while preserving (or even improving) the correlation of geometric embedding similarities with human word similarity judgments. While embedding models have achieved impressive , context distributions capture more semantic features than just word similarity. The distributional inclusion hypothesis (DIH) BID49 BID11 BID6 posits that the context set of a word tends to be a subset of the contexts of its hypernyms. For a concrete example, most adjectives that can be applied to poodle can also be applied to dog, because dog is a hypernym of poodle. For instance, both can be obedient. However, the converse is not necessarily true -a dog can be straight-haired but a poodle cannot. Therefore, dog tends to have a broader context set than poodle. Many asymmetric scoring functions comparing SBOW based on DIH have been developed for automatic hypernymy detection BID49 BID11 BID38.Hypernymy detection plays a key role in many challenging NLP tasks, such as textual entailment BID34, coreference BID32, relation extraction BID8 and question answering BID13. Leveraging the variety of contexts and inclusion properties in context distributions can greatly increase the ability to discover taxonomic structure among words BID38. The inability to preserve these features limits the semantic representation power and downstream applicability of some popular existing unsupervised learning approaches such as word2vec. Several recently proposed methods aim to encode hypernym relations between words in dense embeddings, such as Gaussian embedding BID45 BID0, order embedding BID44, H-feature detector BID33, HyperScore , dual tensor BID12, Poincaré embedding BID28, and LEAR BID46. However, the methods focus on supervised or semi-supervised setting BID44 BID33 BID27 BID12 BID46, do not learn from raw text BID28 or lack comprehensive experiments on the hypernym detection task BID45 BID0.Recent studies BID21 BID38 have underscored the difficulty of generalizing supervised hypernymy annotations to unseen pairs -classifiers often effectively memorize prototypical hypernyms ('general' words) and ignore relations between words. These findings motivate us to develop more accurate and scalable unsupervised embeddings to detect hypernymy and propose several scoring functions to analyze the embeddings from different perspectives. • A novel unsupervised low-dimensional embedding method to model inclusion relations among word contexts via performing non-negative matrix factorization (NMF) on a weighted PMI matrix, which can be efficiently optimized using modified skip-grams.• Several new asymmetric comparison functions to measure inclusion and generality properties and to evaluate different aspects of unsupervised embeddings.• Extensive experiments on 11 datasets demonstrate the learned embeddings and comparison functions achieve state-of-the-art performances on unsupervised hypernym detection while requiring much less memory and compute than approaches based on the full SBOW.• A qualitative experiment illustrates DIVE can be used to solve word sense disambiguation, especially when efficiently modeling word senses at multiple granularities is desirable. The distributional inclusion hypothesis (DIH) suggests that the context set of a hypernym tends to contain the context set of its hyponyms. That is, when representing a word as the counts of contextual co-occurrences, the count in every dimension of hypernym y tends to be larger than or equal to the corresponding count of its hyponym x: DISPLAYFORM0 where x y means y is a hypernym of x, V is the set of vocabulary, and #(x, c) indicates the number of times that word x and its context word c co-occur in a small window with size |W | in corpus D.Our goal is to produce lower-dimensional embeddings that preserve the inclusion property that the embedding of hypernym y is larger than or equal to the embedding of its hyponym x in every dimension. Formally, the desirable property can be written as DISPLAYFORM1 where d 0 is number of dimensions in the embedding space. We add additional non-negativity constraints, i.e. x[i] ≥ 0, y[i] ≥ 0, ∀i, in order to increase the interpretability of the embeddings (the reason will be explained later in this section). This is a challenging task. In reality, there are a lot of noise and systematic biases which cause the violation of DIH in Equation (i.e. #(x, c) > #(y, c) for some neighboring word c), but the general trend can be discovered by processing several thousands of neighboring words in SBOW together. After the compression, the same trend has to be estimated in a much smaller embedding space which discards most of the information in SBOW, so it is not surprising to see most of the unsupervised hypernymy detection studies use SBOW BID38 and the existing unsupervised embeddings like Gaussian embedding have degraded accuracy BID47. Popular methods of unsupervised word embedding are usually based on matrix factorization BID20. The approaches first compute a co-occurrence statistic between the wth word and the cth context word as the (w, c)th element of the matrix M [w, c]. Next, the matrix M is factorized such that M [w, c] ≈ w T c, where w is the low dimension embedding of wth word and c is the cth context embedding. The statistic in M [w, c] is usually related to pointwise mutual information: P M I(w, c) = log(P (w,c) P (w)·P (c) ), where P (w, c) = #(w,c) |D|, |D| = w∈V c∈V #(w, c) is number of co-occurrence word pairs in the corpus, P (w) = #(w) |D|, #(w) = c∈V #(w, c) is the frequency of the word w times the window size |W |, and similarly for P (c). For example, M [w, c] could be set as positive PMI (PPMI), max(P M I(w, c), 0), or shifted PMI, P M I(w, c) − log(k), like skip-grams with negative sampling (SGNS) BID20. Intuitively, since M [w, c] ≈ w T c, larger embedding values of w at every dimension seems to imply larger w T c, larger M [w, c], larger P M I(w, c), and thus larger co-occurrence count #(w, c). However, the derivation has two flaws: c could be negative and lower #(w, c) could still lead to larger P M I(w, c) as long as the #(w) is small enough. To preserve DIH, we propose a novel word embedding method, distributional inclusion vector embedding (DIVE), which fixes the two flaws by performing non-negative factorization (NMF) on the matrix M, where DISPLAYFORM0 where k is a constant which shifts PMI value like SGNS, Z = |D| |V | is the average word frequency, and |V | is the vocabulary size. The design encourages the inclusion property in DIVE (i.e. Equation) to be satisfied because the property implies that Equation (DIH) holds if the matrix is reconstructed perfectly. The derivation is simple: Since context vector c is non-negative, if the embedding of hypernym y is greater than or equal to the embedding of its hyponym x in every dimension, DISPLAYFORM1 and only #(w, c) change with w. Due to its appealing scalability properties during training time BID20, we optimize our embedding based on the skip-gram with negative sampling (SGNS) BID22. The objective function of SGNS is DISPLAYFORM0 where w ∈ R, c ∈ R, c N ∈ R, k is a constant hyper-parameter indicating the ratio between positive and negative samples. BID18 prove SGNS is equivalent to factorizing a shifted PMI matrix M, where (w) and applying non-negativity constraints to the embeddings, DIVE can be optimized using the similar objective function: DISPLAYFORM1 DISPLAYFORM2 where w ≥ 0, c ≥ 0, c N ≥ 0, σ is the logistic sigmoid function, and k is a constant hyper-parameter. P D is the distribution of negative samples, which we set to be the corpus word frequency distribution in this paper. Equation FORMULA6 is optimized by ADAM BID15, a variant of stochastic gradient descent (SGD). The non-negativity constraint is implemented by projection (i.e., clipping any embedding which crosses the zero boundary after an update). The optimization process provides an alternative angle to explain how DIVE preserves DIH. The gradients for the word embedding w is DISPLAYFORM3 Assume hyponym x and hypernym y satisfy DIH in Equation and the embeddings x and y are the same at some point during the gradient ascent. In the case, the gradients coming from negative sampling (the second term) decrease the same amount of embedding values for both x and y because k is a constant hyper-parameter. However, the embedding of hypernym y would get higher or equal positive gradients from the first term than x in every dimension because #(x, c) ≤ #(y, c). This means Equation tends to imply Equation. Combining the analysis from the matrix factorization viewpoint, DIH in Equation FORMULA0 is approximately equivalent to the inclusion property in DIVE (i.e. Equation FORMULA1). For a frequent target word, there must be many neighboring words that incidentally appear near the target word without being semantically meaningful, especially when a large context window size is used. The unrelated context words cause noise in both the word vector and the context vector of DIVE. We address this issue by filtering out context words c for each target word w when the PMI of the co-occurring words is too small (i.e., log( DISPLAYFORM0 . That is, we set #(w, c) = 0 in the objective function. This preprocessing step is similar with computing PPMI in SBOW BID5, where low PMI co-occurrences are removed from the count-based representation. After applying the non-negativity constraint, we observe that each dimension roughly corresponds to a topic, as previous findings suggest BID29 BID24. This gives rise to a natural and intuitive interpretation of our word embeddings: the word embeddings can be seen as unnormalized probability distributions over topics. By removing the normalization of the target word frequency in the shifted PMI matrix, specific words have values in few dimensions (topics), while general words appear in more topics and correspondingly have high values in more dimensions, so the concreteness level of two words can be easily compared using the magnitude of their embeddings. In other words, general words have more diverse context distributions, so we need more dimensions to store the information in order to compress SBOW well BID25.In FIG0, we present three mentions of the word core and its surrounding contexts. These various context words increase the embedding values in different dimensions. Each dimension of the learned embeddings roughly corresponds to a topic, and the more general or representative words for each topic tend to have the higher value in the corresponding dimension (e.g. words in the second column of the table). The embedding is able to capture the common contexts where the word core appears. For example, the context of the first mention is related to the atom topic (dimension id 1) and the electron topic (id 9), while the second and third mention occur in the computer architecture topic (id 2) and education topic (id 11), respectively. We describe four experiments in Section 4-7. The first 3 experiments compare DIVE with other unsupervised embeddings and SBOW using different hypernymy scoring functions. In these experiments, unsupervised approaches refer to the methods that only train on plaintext corpus without using any hypernymy or lexicon annotation. The last experiment presents qualitative on word sense disambiguation. The SBOW and embeddings are tested on 11 datasets. The first 4 datasets come from the recent review of BID38: BLESS BID1, EVALution BID36, Lenci/Benotto , and Weeds BID50. The next 4 datasets are downloaded from the code repository of the H-feature detector BID33: Medical (i.e., Levy 2014), LEDS (also referred to as ENTAILMENT or Baroni 2012) BID3, TM14 (i.e., Turney 2014) BID43, and Kotlerman 2010 BID16. In addition, the performance on the test set of HyperNet BID40 ) (using the random train/test split), the test set of WordNet BID44, and all pairs in HyperLex BID47 are also evaluated. The F1 and accuracy measurements are sometimes very similar even though the quality of prediction varies, so average precision AP@all is adopted as the main evaluation metric. The HyperLex dataset has a continuous score on each candidate word pair, so we adopt Spearman rank coefficient ρ as suggested by the review study of BID47. Any OOV (out-of-vocabulary) word encountered in the testing data is pushed to the bottom of the prediction list (effectively assuming the word pair does not have a hypernym relation). We use WaCkypedia corpus BID2 ), a 2009 Wikipedia dump, to compute SBOW and train the embedding. For the datasets without Part of Speech (POS) information (i.e. Medical, LEDS, TM14, Kotlerman 2010, and HyperNet), the training data of SBOW and embeddings are raw text. For other datasets, we concatenate each token with the Part of Speech (POS) of the token before training the models except the case when we need to match the training setup of another paper. All words are lower cased. Stop words and rare words (occurs less than 10 times) are removed during our preprocessing step. The number of embedding dimensions in DIVE d 0 is set to be 100. Other hyper-parameters used in the experiments are listed in the supplementary materials. The hyper-parameters of DIVE were decided based on the performance of HyperNet training set. To train embeddings more efficiently, we chunk the corpus into subsets/lines of 100 tokens instead of using sentence segmentation. Preliminary experiments show that this implementation simplification does not hurt the performance. In the following experiments, we train both SBOW and DIVE on only the first 512,000 lines (51.2 million tokens) because we find this way of training setting provides better performances (for both SBOW and DIVE) than training on the whole WaCkypedia or training on randomly sampled 512,000 lines. We suspect this is due to the corpus being sorted by the Wikipedia page titles, which makes some categorical words such as animal and mammal occur 3-4 times more frequently in the first 51.2 million tokens than the rest. The performances of training SBOW PPMI on the whole WaCkypedia is also provided for reference in TAB6. If a pair of words has the hypernym relation, the words tend to be similar and the hypernym should be more general than the hyponym. As in HyperScore , we score the hypernym candidates by multiplying two factors corresponding to these properties. The C·∆S (i.e. the cosine similarity multiply the difference of summation) scoring function is defined as DISPLAYFORM0 where w p is the embedding of hypernym and w q is the embedding of hyponym. As far as we know, Gaussian embedding (GE) is the only unsupervised embedding method which can capture the asymmetric relations between a hypernym and its hyponyms. Using the same training and testing setup, we use the code implemented by BID0 1 to train Gaussian embedding on the first 51.2 million tokens and test the embeddings on 11 datasets. Its hyper-parameters are determined using the same way as DIVE (i.e. maximizing the AP on HyperNet training set). We compare DIVE with GE 2 in TAB1, and the performances of random scores and only measuring word similarity using skip-grams are also presented for reference. As we can see, DIVE is usually significantly better than other baselines. In Experiment 1, we show that there exists a scoring function (C·∆S) which detects hypernymy accurately using the embedding space of DIVE. Nevertheless, different scoring functions measure different signals in SBOW or embeddings. Since there are so many scoring functions and datasets available in the domain, we first introduce and test the performances of various scoring functions so as to select the representative ones for a more comprehensive evaluation of DIVE on the hypernymy detection tasks. We denote the embedding/context vector of the hypernym candidate and the hyponym candidate as w p and w q, respectively. The SBOW model which represents a word by the frequency of its neighboring words is denoted as SBOW Freq, while the SBOW which uses PPMI of its neighboring words as the features BID5 ) is denoted as SBOW PPMI. A hypernym tends to be similar to its hyponym, so we measure the cosine similarity between word vectors of the SBOW features BID21 or DIVE. We refer to the symmetric scoring function as Cosine or C for short in the following tables. We also train the original skip-grams with 100 dimensions and measure the cosine similarity between the ing word2vec embeddings. This scoring function is referred to as Word2vec or W. The distributional informativeness hypothesis BID35 observes that in many corpora, semantically'general' words tend to appear more frequently and in more varied contexts. Thus, BID35 advocate using entropy of context distributions to capture the diversity of context. We adopt the two variations of the approach proposed by BID38: SLQS Row and SLQS Sub functions. We also refer to SLQS Row as ∆E because it measures the entropy difference of context distributions. For SLQS Sub, the number of top context words is fixed as 100.Although effective at measuring diversity, the entropy totally ignores the frequency signal from the corpus. To leverage the information, we measure the generality of a word by its L1 norm (|w p | 1) and L2 norm (||w p || 2). Recall that Equation indicates that the embedding of the hypernym y should have a larger value at every dimension than the embedding of the hyponym x. When the inclusion property holds, |y| 1 = i y[i] ≥ i x[i] = |x| 1 and similarly ||y|| 2 ≥ ||x|| 2. Thus, we propose two scoring functions, difference of vector summation (|w p | 1 − |w q | 1) and the difference of vector 2-norm (||w p || 2 − ||w q || 2). Notice that when applying the difference of vector summations (denoted as ∆S) to SBOW Freq, it is equivalent to computing the word frequency difference between the hypernym candidate pair. The combination of 2 similarity functions (Cosine and Word2vec) and the 3 generality functions (difference of entropy, summation, and 2-norm of vectors) leads to six different scoring functions as shown in TAB2, and C·∆S is the same scoring function we used in Experiment 1. It should be noted that if we use skip-grams with negative sampling (word2vec) as the similarity measurement (i.e., W · ∆ {E,S,Q}), the scores are determined by two embedding/feature spaces together (word2vec and DIVE/SBOW). Several scoring functions are proposed to measure inclusion properties of SBOW based on DIH. Weeds Precision BID49 and CDE BID7 ) both measure the magnitude of the intersection between feature vectors (|w p ∩ w q |). For example, w p ∩ w q is defined by the elementwise minimum in CDE. Then, both scoring functions divide the intersection by the magnitude of the potential hyponym vector (|w q |). invCL BID17 ) (A variant of CDE) is also tested. We choose these 3 functions because they have been shown to detect hypernymy well in a recent study BID38. However, it is hard to confirm that their good performances come from the inclusion property between context distributions -it is also possible that the context vectors of more general words have higher chance to overlap with all other words due to their high frequency. For instance, considering a one dimension feature which stores only the frequency of words, the naive embedding could still have reasonable performance on the CDE function, but the embedding DISPLAYFORM0 where w 0 is a constant which emphasizes the inclusion penalty. If w 0 = 1 and a = 1, AL 1 is equivalent to L1 distance. The lower AL 1 distance implies a higher chance of observing the hypernym relation. We tried w 0 = 5 and w 0 = 20. w 0 = 20 produces a worse micro-average AP@all on SBOW Freq, SBOW PPMI and DIVE, so we fix w 0 to be 5 in all experiments. An efficient way to solve the optimization in AL 1 is presented in the supplementary materials. We show the micro average AP@all on 10 datasets using different hypernymy scoring functions in TAB2. We can see the combination functions such as C·∆S and W·∆S perform the best overall. Among the unnormalized inclusion based scoring functions, CDE works the best. AL 1 performs well compared with other functions which remove the frequency signal such as Word2vec, Cosine, and SLQS Row. The summation is the most robust generality measurement. In the In TAB4, DIVE with two of the best scoring functions (C·∆S and W·∆S) is compared with the previous unsupervised state-of-the-art approaches based on SBOW on different datasets. There are several reasons which might cause the large performance gaps in some datasets. In addition to the effectiveness of DIVE, some improvements come from our proposed scoring functions. The fact that every paper uses a different training corpus also affects the performances. Furthermore, BID38 select the scoring functions and feature space for the first 4 datasets based on AP@100, which we believe is too sensitive to the hyper-parameter settings of different methods. To isolate the impact of each factor, we perform a more comprehensive comparison next. In this experiment, we examine whether DIVE successfully preserves the signals for hypernymy detection tasks, which are measured by the same scoring functions designed for SBOW. Summation difference (∆S) and CDE perform the best among generality and inclusion functions in TAB2, BID17, APSyn BID37, and CDE BID7 ) are selected because they have the best AP@100 in the first 4 datasets BID38. Cosine similarity BID21, balAPinc BID16 in 3 datasets BID43, SLQS BID35 in HyperNet dataset BID40, and Freq ratio (FR) BID47 respectively. AL 1 could be used to examine the inclusion properties after removing the frequency signal. Therefore, we will present the using these 3 scoring functions, along with W·∆S and C·∆S. In addition to classic representations such as SBOW Freq and SBOW PPMI, we compare distributional inclusion vector embedding (DIVE) with additional 4 baselines in TAB6.• SBOW PPMI with additional frequency weighting (PPMI w/ FW). Specifically, w[c] = max(log( DISPLAYFORM0), 0). This forms the matrix reconstructed by DIVE when k = 1.• DIVE without the PMI filter (DIVE w/o PMI) • NMF on shifted PMI: Non-negative matrix factorization (NMF) on the shifted PMI without frequency weighting for DIVE (DIVE w/o FW). This is the same as applying the nonnegative constraint on the skip-gram model. • K-means (Freq NMF): The method first uses Mini-batch k-means BID39 to cluster words in skip-gram embedding space into 100 topics, and hashes each frequency count in SBOW into the corresponding topic. If running k-means on skip-grams is viewed as an approximation of clustering the SBOW context vectors, the method can be viewed as a kind of NMF BID9. Let the N × N context matrix be denoted as M c, where the (i, j)th element stores the count of word j appearing beside word i. K-means hashing creates a N × 100 matrix G with orthonormal rows (G T G = I), where the (i, k)th element is 0 if the word i does not belong to cluster k. The orthonormal G is also an approximated solution of a type of NMF (M c ≈ F G T) BID9. Hashing context vectors into topic vectors can be written DISPLAYFORM1 In the experiment, we also tried to apply a constant log(k) shifting to SBOW PPMI (i.e. max(P M I − log(k), 0)). We found that the performance degrades as k increases. Similarly, applying PMI filter to SBOW PPMI (set context feature to be 0 if the value is lower than log(k f)) usually makes the performances worse, especially when k f is large. Applying PMI filter to SBOW Freq only makes its performances closer to (but still much worse than) SBOW PPMI, so we omit this baseline as well. In TAB6, we first confirm the finding of the previous review study of BID38: there is no single hypernymy scoring function which always outperforms others. One of the main reasons is that different datasets collect negative samples differently. This is also why we evaluate our method on many datasets to make sure our hold in general. For example, if negative samples come from random word pairs (e.g. WordNet dataset), a symmetric similarity measure is already a pretty good scoring function. On the other hand, negative samples come from related or similar words in HyperNet, EVALution, Lenci/Benotto, and Weeds, so only computing generality difference leads to the best (or close to the best) performance. The negative samples in many datasets are composed of both random samples and similar words (such as BLESS), so the combination of similarity and generality difference yields the most stable . DIVE performs similar or better on all the scoring functions compared with SBOW consistently across all datasets in TAB6, while using many fewer dimensions (see TAB8). Its on combination scoring functions outperform SBOW Freq. Meanwhile, its on AL 1 outperform SBOW PPMI. The fact that combination scoring functions (i.e., W·∆S or C·∆S) usually outperform generality functions suggests that only memorizing general words is not sufficient. The best average performance on 4 and 10 datasets are both produced by W·∆S on DIVE.SBOW PPMI improves the combination functions from SBOW Freq but sacrifices AP on the inclusion functions. It generally hurts performance to change the frequency sampling of PPMI (PPMI w/ FW) or compute SBOW PPMI on the whole WaCkypedia (all wiki) instead of the first 51.2 million tokens. The similar trend can also be seen in TAB7. Note that AL 1 completely fails in HyperLex dataset using SBOW PPMI, which suggests that PPMI might not necessarily preserve the distributional inclusion property, even though it can have good performance on combination functions. Removing the PMI filter from DIVE slightly drops the overall precision while removing frequency weights on shifted PMI (w/o FW) leads to poor performances. K-means (Freq NMF) produces similar AP compared with SBOW Freq, but has worse AL 1 scores. Its best AP scores on different datasets are also significantly worse than the best AP of DIVE. This means that only making word2vec (skip-grams with negative sampling) non-negative or naively accumulating topic distribution in contexts cannot lead to satisfactory embeddings. In addition to hypernymy detection, BID0 show that the mixture of Gaussian distributions can also be used to discover multiple senses of each word. In our qualitative experiment, we show that DIVE can achieve the similar goal without fixing the number of senses before training the embedding. Recall that each dimension roughly corresponds to one topic. Given a query word, the higher embedding value on a dimension implies higher likelihood to observe the word in the context of the topic. The embedding of a polysemy would have high values on different groups of topics/dimensions. This allows us to discover the senses by clustering the topics/dimensions of the polysemy. We use the embedding values as the feature each dimension, compute the pairwise similarity between dimensions, and apply spectral clustering BID41 to group topics as shown in the TAB9. See more implementation details in the supplementary materials. In the word sense disambiguation tasks, it is usually challenging to determine how many senses/clusters each word should have. Many existing approaches fix the number of senses before training the embedding BID42 BID0. BID26 make the number of clusters approximately proportional to the diversity of the context, but the assumption does not always hold. Furthermore, the training process cannot capture different granularity of senses. For instance, race in the car context could share the same sense with the race in the game topic because they all mean contest, but the race in the car context actually refers to the specific contest of speed. Therefore, they can also be viewed as separate senses (like the in TAB9). This means the correct number of clusters is not unique, and the methods, which fixes the cluster numbers, need to re-train the embedding many times to capture such granularity. In our approach, clustering dimensions is done after the training process of DIVE is completed, so it is fairly efficient to change the cluster numbers and hierarchical clustering is also an option. Similar to our method, BID31 also discover word senses by graph-based clustering. The main difference is that they cluster the top n words which are most related to the query word instead of topics. However, choosing the hyper-parameter n is difficult. Large n would make graph clustering algorithm inefficient, while small n would make less frequent senses difficult to discover. Most previous unsupervised approaches focus on designing better hypernymy scoring functions for sparse bag of word (SBOW) features. They are well summarized in the recent study BID38. BID38 also evaluate the influence of different contexts, such as changing the window size of contexts or incorporating dependency parsing information, but neglect scalability issues inherent to SBOW methods. A notable exception is the Gaussian embedding model BID45. The context distribution of each word is encoded as a multivariate Gaussian distribution, where the embeddings of hypernyms tend to have higher variance and overlap with the embedding of their hyponyms. However, since a Gaussian distribution is normalized, it is difficult to retain frequency information during the embedding process, and experiments on HyperLex BID47 demonstrate that a simple baseline only relying on word frequency can achieve good . Follow-up work models contexts by a mixture of Gaussians BID0 relaxing the unimodality assumption but achieves little improvement on hypernym detection tasks. BID14 show that images retrieved by a search engine can be a useful source of information to determine the generality of lexicons, but the resources might not be available for some corpora such as scientific literature. Order embedding BID44 ) is a supervised approach to encode many annotated hypernym pairs (e.g. all of the whole WordNet BID23) into a compact embedding space, where the embedding of a hypernym should be smaller than the embedding of its hyponym in every dimension. Our method learns embedding from raw text, where a hypernym embedding should be larger than the embedding of its hyponym in every dimension. Thus, DIVE can be viewed as an unsupervised and reversed form of order embedding. Other semi-supervised hypernym detection methods aim to generalize from sets of annotated word pairs using raw text corpora. The goal of HyperScore BID27 is similar to our model: the embedding of a hypernym should be similar to its hyponym but with higher magnitude. However, their training process relies heavily on annotated hypernym pairs, and the performance drops significantly when reducing the amount of supervision. In addition to context distributions, previous work also leverages training data to discover useful text pattern indicating is-a relation BID40 BID33, but it remains challenging to increase recall of hypernym detection because commonsense facts like cat is-a animal might not appear in the corpus. Non-negative matrix factorization (NMF) has a long history in NLP, for example in the construction of topic models BID29. Non-negative sparse embedding (NNSE) BID24 and BID10 indicate that non-negativity can make embeddings more interpretable and improve word similarity evaluations. The sparse NMF is also shown to be effective in cross-lingual lexical entailment tasks but does not necessarily improve monolingual hypernymy detection BID48. In our study, a new type of NMF is proposed, and the comprehensive experimental analysis demonstrates its state-of-the-art performances on unsupervised hypernymy detection. Compressing unsupervised SBOW models into a compact representation is challenging while preserving the inclusion, generality, and similarity signals which are important for hypernym detection. Our experiments suggest that simple baselines such as accumulating K-mean clusters and non-negative skip-grams do not lead to satisfactory performances in this task. To achieve this goal, we proposed an interpretable and scalable embedding method called distributional inclusion vector embedding (DIVE) by performing non-negative matrix factorization (NMF) on a weighted PMI matrix. We demonstrate that scoring functions which measure inclusion and generality properties in SBOW can also be applied to DIVE to detect hypernymy, and DIVE performs the best on average, slightly better than SBOW while using many fewer dimensions. Our experiments also indicate that unsupervised scoring functions, which combine similarity and generality measurements, work the best in general, but no one scoring function dominates across all datasets. A combination of unsupervised DIVE with the proposed scoring functions produces new state-of-the-art performances on many datasets under the unsupervised setup. Finally, a qualitative experiment shows that clusters of the topics discovered by DIVE often correspond to the word senses, which allow us to do word sense disambiguation without the need to know the number of senses before training the embeddings. In addition to the unsupervised approach, we also compare DIVE with semi-supervised approaches. When there are sufficient training data, there is no doubt that the semi-supervised embedding approaches such as HyperNet BID40, H-feature detector BID33, and HyperScore can achieve better performance than all unsupervised methods. However, in many domains such as scientific literature, there are often not many annotated hypernymy pairs (e.g. Medical dataset).Since we are comparing an unsupervised method with semi-supervised methods, it is hard to fairly control the experimental setups and tune the hyper-parameters. In TAB10, we only show several performances which are copied from the original paper when training data are limited 3. As we can see, the performance from DIVE is roughly comparable to the previous semi-supervised approaches trained on small amount of hypernym pairs. This demonstrates the robustness of our approach and the difficulty of generalizing hypernymy annotations with semi-supervised approaches. In TAB11, we show the most general words in DIVE under different queries as constraints. We also present the accuracy of judging which word is a hypernym (more general) given word pairs with hypernym relations in TAB1. The direction is classified correctly if the generality score is greater than 0 (hypernym is indeed predicted as the more general word). For instance, summation difference (∆S) classifies correctly if DISPLAYFORM0 From the table, we can see that the simple summation difference performs better than SQLS Sub, and DIVE predicts directionality as well as SBOW. Notice that whenever we encounter OOV, the directionality is predicted randomly. If OOV is excluded, the accuracy of predicting directionality using unsupervised methods can reach around 0.7-0.75. In HyperNet and WordNet, some hypernym relations are determined between phrases instead of words. Phrase embeddings are composed by averaging word embeddings or SBOW features. For WordNet, we assume the Part of Speech (POS) tags of the words are the same as the phrase. All part-of-speech (POS) tags in the experiments come from NLTK.The window size |W | of SBOW, DIVE, and GE are set as 20 (left 10 words and right 10 words). For DIVE, the number of epochs is 15, the learning rate is 0.001, the batch size is 128, the threshold in PMI filter k f is set to be 30, and the ratio between negative and positive samples (k) is 1.5. The hyper-parameters of DIVE were decided based on the performance of HyperNet training set. The window size of skip-grams (word2vec) is 10. The number of negative samples (k) in skip-gram is set as 5. When composing skip gram into phrase embedding, average embedding is used. For Gaussian embedding (GE), the number of mixture is 1, the number of dimension is 100, the learning rate is 0.01, the lowest variance is 0.1, the highest variance is 100, the highest Gaussian mean is 10, and other hyper-parameters are the default value in https://github.com/ benathi/word2gm. The hyper-parameters of GE were also decided based on the performance of HyperNet training set. When determining the score between two phrases, we use the average score of every pair of tokens in two phrases. The number of testing pairs N and the number of OOV word pairs is presented in TAB1. We use all the default hyper-parameters of the spectral clustering library in Scikit-learn 0. However, clustering on the global features might group topics together based on the co-occurrence of words which are unrelated to the query words and we want to make the similarity dependent on the query word. For example, a country topic should be clustered together with a city topic if the query word is place, but it makes more sense to group the country topic with the money topic together if the query word is bank like we did in the word sense disambiguation experiment TAB9. This means we want to focus on the geographical meaning of country when the query is related to geography, while focus on the economic meaning of country when the query is about economics. To create query dependent similarity measurement, we only consider the embedding of words which are related to the query word when preparing the features of dimensions. Specifically, given a query word q, the feature vector of the ith dimension f (c i, q) is defined as: DISPLAYFORM0 where w q [c j] is the value of jth dimension of query word embedding, C j (n) is the set of embeddings of top n words in the jth dimension, and the operator ⊕ means concatenation. This means instead of considering all the words in the vocabulary, we only take the top n words of every dimension j (n is fixed as 100 in the experiment), weight the feature based on how likely to observe query word in dimension j (w q [c j]), and concatenate all features together. That is, when measuring the similarity of dimensions, we only consider the aspects related to query word (e.g. mostly considering words related to facility and money when the query word is bank).After the features of all dimensions are collected, we normalize the feature of each dimension to have the norm 1, compute the pairwise similarity and run the spectral clustering to get the clustering . A.5 EFFICIENT WAY TO COMPUTE ASYMMETRIC L1 (AL 1)Recall that Equation FORMULA10 FIG1, an simple example is visualized to illustrate the intuition behind the distance function. By adding slack variables ζ and ξ, the problem could be converted into a linear programming problem: dq a*dq dp a*dq-dp dp-a*dq

Represent the short summary for finding the corresponding scientific article (long)

We propose a novel unsupervised word embedding which preserves the inclusion property in the context distribution and achieve state-of-the-art results on unsupervised hypernymy detection

We propose a new variational hashing-based collaborative filtering approach optimized for a novel self-mask variant of the Hamming distance, which outperforms state-of-the-art by up to 12% on NDCG.


Question: Represent the text to find a very short summary (1-2 sentences max).

Question: Continual learning aims to learn new tasks without forgetting previously learned ones. This is especially challenging when one cannot access data from previous tasks and when the model has a fixed capacity. Current regularization-based continual learning algorithms need an external representation and extra computation to measure the parameters' \textit{importance}. In contrast, we propose Uncertainty-guided Continual Bayesian Neural Networks (UCB) where the learning rate adapts according to the uncertainty defined in the probability distribution of the weights in networks. Uncertainty is a natural way to identify \textit{what to remember} and \textit{what to change} as we continually learn, and thus mitigate catastrophic forgetting. We also show a variant of our model, which uses uncertainty for weight pruning and retains task performance after pruning by saving binary masks per tasks. We evaluate our UCB approach extensively on diverse object classification datasets with short and long sequences of tasks and report superior or on-par performance compared to existing approaches. Additionally, we show that our model does not necessarily need task information at test time, i.e.~it does not presume knowledge of which task a sample belongs to. Humans can easily accumulate and maintain knowledge gained from previously observed tasks, and continuously learn to solve new problems or tasks. Artificial learning systems typically forget prior tasks when they cannot access all training data at once but are presented with task data in sequence. Overcoming these challenges is the focus of continual learning, sometimes also referred to as lifelong learning or sequential learning. Catastrophic forgetting refers to the significant drop in the performance of a learner when switching from a trained task to a new one. This phenomenon occurs because trained parameters on the initial task change in favor of learning new objectives. This is the reason that naive finetuning intuitively suffers from catastrophic forgetting. Given a network of limited capacity, one way to address this problem is to identify the importance of each parameter and penalize further changes to those parameters that were deemed to be important for the previous tasks a; ). An alternative is to freeze the most important parameters and allow future tasks to only adapt the remaining parameters to new tasks . Such models rely on the explicit parametrization of importance. We propose here implicit uncertainty-guided importance representation. Bayesian approaches to neural networks (b) can potentially avoid some of the pitfalls of explicit parameterization of importance in regular neural networks. Bayesian techniques, naturally account for uncertainty in parameters estimates. These networks represent each parameter with a distribution defined by a mean and variance over possible values drawn from a shared latent probability distribution . Variational inference can approximate posterior distributions using Monte Carlo sampling for gradient estimation. These networks act like ensemble methods in that they reduce the prediction variance but only use twice the number of parameters present in a regular neural network. We propose the use of the predicted mean and variance of the latent distributions to characterize the importance of each parameter. We perform continual learning with Bayesian neural networks by controlling the learning rate of each parameter as a function of its uncertainty. Figure 1 illustrates how posterior distributions evolve for certain and uncertain weight Illustration of evolution of weight distributions through learning two tasks. (a) circles represent weight parameters, initialized by distributions with mean and variance values randomly sampled from Ɲ(0,0.1). As an example we show five color-coded and plot their distributions. (b) Shows posterior distribution after learning Task 1. While W1 and W2 exhibit lower uncertainties (more contributions in learning Task 1), W3, W4, and W5 appear to have larger uncertainties, with the highest STD in W5, making them available to learn more tasks. (c) Task 2 is learned using higher learning rates for previously uncertain parameters (W3 and W4, W5) while learning rates for W1 and W2 are moderated according to their predicted low uncertainty after finishing task 1. Figure 1: Illustration of the evolution of weight distributions -uncertain weights adapt more quicklywhen learning two tasks using UCB. (a) weight parameter initialized by distributions initialized with mean and variance values randomly sampled from N (0, 0.1). (b) posterior distribution after learning task 1; while θ 1 and θ 2 exhibit lower uncertainties after learning the first task, θ 3, θ 4, and θ 5 have larger uncertainties, making them available to learn more tasks. (c) a second task is learned using higher learning rates for previously uncertain parameters (θ 1, θ 2, θ 3, and θ 4) while learning rates for θ 1 and θ 2 are reduced. Size of the arrows indicate the magnitude of the change of the distribution mean upon gradient update. distributions while learning two consecutive tasks. Intuitively, the more uncertain a parameter is, the more learnable it can be and therefore, larger gradient steps can be taken for it to learn the current task. As a hard version of this regularization technique, we also show that pruning, i.e., preventing the most important model parameters from any change and learning new tasks with the remaining parameters, can be also integrated into UCB. We refer to this method as UCB-P. We propose to perform continual learning with Bayesian neural networks and develop a new method which exploits the inherent measure of uncertainty therein to adapt the learning rate of individual parameters (Sec. 4). Second, we introduce a hard-threshold variant of our method that decides which parameters to freeze (Sec. 4.2). Third, in Sec. 5, we extensively validate our approach experimentally, comparing it to prior art both on single datasets split into different tasks, as well as for the more difficult scenario of learning a sequence of different datasets. Forth, in contrast to most prior work, our approach does not rely on knowledge about task boundaries at inference time, which humans do not need and might not be always available. We show in Sec. 6 that our approach naturally supports this scenario and does not require task information at test time, sometimes also referred to as a "single head" scenario for all tasks. We refer to evaluation metric of a "single head" model without task information at test time as "generalized accuracy". Our code will be released upon acceptance. Conceptually, approaches to continual learning can be divided into the following categories: dynamic architectural methods, memory-based methods, and regularization methods. In this setting, the architecture grows while keeping past knowledge fixed and storing new knowledge in different forms such as additional layers, nodes, or modules. In this approach, the objective function remains fixed whereas the model capacity grows -often exponentially-with the number of tasks. Progressive networks ) was one of the earliest works in this direction and was successfully applied to reinforcement learning problems; the base architecture was duplicated and lateral connections added in response to new tasks. Dynamically Expandable Network (DEN) also expands its network by selecting drifting units and retraining them on new tasks. In contrast to our method, these approaches require the architecture grow with each new task. Memory-based methods: In this regime, previous information is partially stored to be used later as a form of rehearsal . Gradient episodic memory (GEM) uses this idea to store the data at the end of each episode to be used later to prevent gradient updates from deviating from their previous values. GEM also allows for positive backward knowledge transfer, i.e, an improvement on previously learned tasks, and it was the first method capable of learning using a single training example. Recent approaches in this category have mitigated forgetting by using external data combined with distillation loss and/or confidence-based sampling strategies to select the most representative samples. (; ;) Regularization methods: In these approaches, significant changes to the representation learned for previous tasks are prevented. This can be performed through regularizing the objective function or directly enforced on weight parameters. Typically, this importance measure is engineered to represent the importance of each parameter. Inspired by Bayesian learning, in elastic weight consolidation (EWC) method important parameters are those to have the highest in terms of the Fisher information matrix. In Synaptic Intelligence (SI) this parameter importance notion is engineered to correlate with the loss function: parameters that contribute more to the loss are more important. Similar to SI, Memory-aware Synapses (MAS) (a) proposed an online way of computing importance adaptive to the test set using the change in the model outputs w.r.t the inputs. While all the above algorithms are task-dependent, in parallel development to this work, (b) has recently investigated task-free continual learning by building upon MAS and using a protocol to update the weights instead of waiting until the tasks are finished. PackNet used iterative pruning to fully restrict gradient updates on important weights via binary masks. This method requires knowing which task is being tested to use the appropriate mask. PackNet also ranks the weight importance by their magnitude which is not guaranteed to be a proper importance indicative. HAT identifies important neurons by learning an attention vector to the task embedding to control the gradient propagation. It maintains the information learned on previous tasks using an almost-binary mask per previous tasks. Bayesian approaches: Using Bayesian approach in learning neural networks has been studied for few decades (b; a). Several approaches have been proposed for Bayesian neural networks, based on, e.g., the Laplace approximation (a), Hamiltonian Monte Carlo , variational inference , and probabilistic backpropagation (Hernández-). Variational continual learning uses Bayesian inference to perform continual learning where new posterior distribution is simply obtained by multiplying the previous posterior by the likelihood of the dataset belonging to the new task. They also showed that by using a core-set, a small representative set of data from previous tasks, VCL can experience less forgetting. In contrast, we rely on Bayesian neural networks to use their predictive uncertainty to perform continual learning. Moreover, we do not use episodic memory or any other way to access or store previous data in our approach. Natural gradient descent methods: A fast natural gradient descent method for variational inference was introduced in in which, the Fisher Information matrix is approximated using the generalized Gauss-Newton method. In contrast, in our work, we use classic gradient descent. Although second order optimization algorithms are proven to be more accurate than the first order methods, they add considerable computational cost.; both investigate the effect of natural gradient descent methods as an alternative to classic gradient descent used in VCL and EWC methods. GNG uses Gaussian natural gradients in the Adam optimizer in the framework of VCL because as opposed to conventional gradient methods which perform in Euclidian space, natural gradients cause a small difference in terms of distributions following the changes in parameters in the Riemannian space. Similar to VCL, they obtained their best performance by adding a coreset of previous examples. introduce two modifications to VCL called Natural-VCL (N-VCL) and VCL-Vadam. N-VCL uses a Gauss-Newton approximation introduced by to estimate the VCL objective function and used natural gradient method proposed in to exploit the Riemannian geometry of the variational posterior by scaling the gradient with an adaptive learning rate equal to σ −2 obtained by approximating the Fisher Information matrix in an online fashion. VCL-Vadam is a simpler version of N-VCL to trade-off accuracy for simplicity which uses Vadam to update the gradients by perturbing the weights with a Gaussian noise using a reparameterization trick and scaling by σ −1 instead of its squared. N-VCL/VCL-Vadam both use variational inference to adapt the learning rate within Adam optimizer at every time step, whereas in our method below, gradient decent is used with constant learning rate during each task where learning rate scales with uncertainty only after finishing a task. We show extensive comparison with state-of-the-art on short and relatively long sequence of vision datasets with Bayesian convolutional neural networks, whereas VCL-Vadam only rely on multi-layer perceptron networks. We also like to highlight that this is the first work which evaluates and shows the working of convolutional Bayesian Neural Networks rather than only fully connected MLP models for continual learning. In this section, we review the Bayes-by-Backprop (BBB) framework which was introduced by ; to learn a probability distribution over network parameters. showed a back-propagation-compatible algorithm which acts as a regularizer and yields comparable performance to dropout on the MNIST dataset. In Bayesian models, latent variables are drawn from a prior density p(w) which are related to the observations through the likelihood p(x|w). During inference, the posterior distribution p(w|x) is computed conditioned on the given input data. However, in practice, this probability distribution is intractable and is often estimated through approximate inference. Markov Chain Monte Carlo (MCMC) sampling has been widely used and explored for this purpose, see for different methods under this category. However, MCMC algorithms, despite providing guarantees for finding asymptotically exact samples from the target distribution, are not suitable for large datasets and/or large models as they are bounded by speed and scalability issues. Alternatively, variational inference provides a faster solution to the same problem in which the posterior is approximated using optimization rather than being sampled from a chain .Variational inference methods always take advantage of fast optimization techniques such as stochastic methods or distributed methods, which allow them to explore data models quickly. See for a complete review of the theory and for more discussion on how to use Bayes by Backprop (BBB) in convolutioal neural networks. Let x ∈ IR n be a set of observed variables and w be a set of latent variables. A neural network, as a probabilistic model P (y|x, w), given a set of training examples D = (x, y) can output y which belongs to a set of classes by using the set of weight parameters w. Variational inference aims to calculate this conditional probability distribution over the latent variables by finding the closest proxy to the exact posterior by solving an optimization problem. We first assume a family of probability densities over the latent variables w parametrized by θ, i.e., q(w|θ). We then find the closest member of this family to the true conditional probability of interest P (w|D) by minimizing the Kullback-Leibler (KL) divergence between q and P which is equivalent to minimizing variational free energy or maximizing the expected lower bound: The objective function can be written as: Eq. 2 can be approximated using N Monte Carlo samples w i from the variational posterior : We assume q(w|θ) to have a Gaussian pdf with diagonal covariance and parametrized by θ = (µ, ρ). A sample weight of the variational posterior can be obtained by sampling from a unit Gaussian and reparametrized by w = µ + σ • where is the noise drawn from unit Gaussian, and • is a pointwise multipliation. Standard deviation is parametrized as σ = log(1 + exp(ρ)) and thus is always positive. For the prior, as suggested by , a scale mixture of two Gaussian pdfs are chosen which are zero-centered while having different variances of σ 2 1 and σ 2 2. The uncertainty obtained for every parameter has been successfully used in model compression and uncertainty-based exploration in reinforcement learning . In this work we propose to use this framework to learn sequential tasks without forgetting using per-weight uncertainties. In this section, we introduce Uncertainty-guided Continual learning approach with Bayesian neural networks (UCB), which exploits the estimated uncertainty of the parameters' posterior distribution to regulate the change in "important" parameters both in a soft way (Section 4.1) or setting a hard threshold (Section 4.2). A common strategy to perform continual learning is to reduce forgetting by regularizing further changes in the model representation based on parameters' importance. In UCB the regularizing is performed with the learning rate such that the learning rate of each parameter and hence its gradient update becomes a function of its importance. As shown in the following equations, in particular, we scale the learning rate of µ and ρ for each parameter distribution inversely proportional to its importance Ω to reduce changes in important parameters while allowing less important parameters to alter more in favor of learning new tasks. The core idea of this work is to base the definition of importance on the well-defined uncertainty in parameters distribution of Bayesian neural networks, i.e., setting the importance to be inversely proportional to the standard deviation σ which represents the parameter uncertainty in the Baysian neural network: We explore different options to set Ω in our ablation study presented in Section A.2 of the appendix, Table 1. We empirically found that Ω µ = 1/σ and not adapting the learning rate for ρ (i.e. Ω ρ = 1) yields the highest accuracy and the least forgetting. The key benefit of UCB with learning rate as the regularizer is that it neither requires additional memory, as opposed to pruning technique nor tracking the change in parameters with respect to the previously learned task, as needed in common weight regularization methods. More importantly, this method does not need to be aware of task switching as it only needs to adjust the learning rates of the means in the posterior distribution based on their current uncertainty. The complete algorithm for UCB is shown in Algorithm 1 with parameter update function given in Algorithm 2. In this section, we introduce a variant of our method, UCB-P, is related to recent efforts in weight pruning in the context of reducing inference computation and network compression . More specifically, weight pruning has been recently used in continual learning , where the goal is to continue learning multiple tasks using a single network's capacity. accomplished this by freeing up parameters deemed to be unimportant to the current task according to their magnitude. Forgetting is prevented in pruning by saving a task-specific binary mask of important vs. unimportant parameters. Here, we adapt pruning to Bayesian neural networks. Specifically, we propose a different criterion for measuring importance: the statistically-grounded uncertainty defined in Bayesian neural networks. Unlike regular deep neural networks, in a BBB model weight parameters are represented by probability distributions parametrized by their mean and standard deviation. Similar to , in order to take into account both mean and standard deviation, we use the signal-to-noise ratio (SNR) for each parameter defined as SNR is a commonly used measure in signal processing to distinguish between "useful" information from unwanted noise contained in a signal. In the context of neural models, the SNR can be thought as an indicative of parameter importance; the higher the SNR, the more effective or important the parameter is to the model predictions for a given task. Algorithm 1 Uncertainty-guided Continual Learning with Bayesian Neural Networks UCB 1: Require Training data for all tasks D = (x, y), µ (mean of posterior), ρ, σ1 and σ2 (std for the scaled mixture Gaussian pdf of prior), π (weighting factor for prior), N (number of samples in a mini-batch), M (Number of minibatches per epoch), initial learning rate (α0) 2: αµ = αρ = α0 3: for every task do 4: repeat 5: ∼ N (0, I) 6: σ = log(1 + exp(ρ)) Ensures σ is always positive 7: w = µ + σ • w = {w1, . . ., wi, . . ., wN} posterior samples of weights 8: µ ← µ − αµ∇LBBB µ 13: ρ ← ρ − αρ∇LBBB ρ 14: until loss plateaus 15: αµ, αρ ← LearningRateUpdate(αµ, αρ, σ, µ) See Algorithm 2 for UCB and 3 for UCB-P 16: end for Algorithm 2 LearningRateUpdate in UCB 1: function LearningRateUpdate(αµ, αρ, σ) 2: for each parameter do 3: Ωµ ← 1/σ 4: Ωρ ← 1 5: αµ ← αµ/Ωµ 6: αρ ← αρ/Ωρ 7: end for 8: end function Algorithm 3 LearningRateUpdate in UCB-P 1: function LearningRateUpdate(αµ, αρ, σ, µ) 2: for each parameter j in each layer l do 3: Ω ← |µ| /σ Signal to noise ratio 4: if Ω[j] ∈ top p% of Ωs in l then 5: αµ = αρ = 0 6: end if 7: end for 8: end function UCB-P, as shown in Algorithms 1 and 3, is performed as follows: for every layer, convolutional or fully-connected, the parameters are ordered by their SNR value and those with the lowest importance are pruned (set to zero). The pruned parameters are marked using a binary mask so that they can be used later in learning new tasks whereas the important parameters remain fixed throughout training on future tasks. Once a task is learned, an associated binary mask is saved which will be used during inference to recover key parameters and hence the exact performance to the desired task. The overhead memory per parameter in encoding the mask as well as saving it on the disk is as follows. Assuming we have n tasks to learn using a single network, the total number of required bits to encode an accumulated mask for a parameter is at max log 2 n bits assuming a parameter deemed to be important from task 1 and kept being encoded in the mask. Datasets: We evaluate our approach in two common scenarios for continual learning: 1) classincremental learning of a single or two randomly alternating datasets, where each task covers only a subset of the classes in a dataset, and 2) continual learning of multiple datasets, where each task is a dataset. We use Split MNIST with 5 tasks (5-Split MNIST) similar to and permuted MNIST for class incremental learning with similar experimental settings as used in ). Furthermore, to have a better understanding of our method, we evaluate our approach on continually learning a sequence of 8 datasets with different distributions using the identical sequence as in, which includes FaceScrub , MNIST, CIFAR100, NotMNIST , SVHN , CIFAR10, TrafficSigns , and FashionMNIST . Details of each are summarized in Table 4 in appendix. No data augmentation of any kind has been used in our analysis. Baselines: Within the Bayesian framework, we compare to three models which do not incorporate the importance of parameters, namely fine-tuning, feature extraction, and joint training. In fine-tuning (BBB-FT), training continues upon arrival of new tasks without any forgetting avoidance strategy. Feature extraction, denoted as (BBB-FE), refers to freezing all layers in the network after training the first task and training only the last layer for the remaining tasks. In joint training (BBB-JT) we learn all the tasks jointly in a multitask learning fashion which serves as the upper bound for average accuracy on all tasks, as it does not adhere to the continual learning scenario. We also perform the counterparts for FT, FE, and JT using ordinary neural networks and denote them as ORD-FT, ORD-FE, and ORD-JT. From the prior work, we compare with state-of-the-art approaches including Elastic Weight Consolidation (EWC), Incremental Moment Matching (IMM), Learning Without Forgetting (LWF) , Less-Forgetting Learning (LFL), PathNet, Progressive neural networks (PNNs), and Hard Attention Mask (HAT) using implementations provided by. On Permuted MNIST for SI are reported from. On Split and Permuted MNIST, for VCL are obtained using their original provided code whereas for VCL-GNG and VCL-Vadam are reported from the original work without re-implementation. Because our method lies into the regularization-based regime, we only compare against baselines which do not benefit from episodic or coreset memory. Hyperparameter tuning: Unlike commonly used tuning techniques which use a validation set composed of all classes in the dataset, we only rely on the first two task and their validations set, similar to the setup in . In all our experiments we consider a 0.15 split for the validation set on the first two tasks. After tuning, training starts from the beginning of the sequence. Our scheme is different from , where the models are trained on the first (e.g. three) tasks for validation and then training is restarted for the remaining ones and the reported performance is only on the remaining tasks. Training details: It is important to note that in all our experiments, no pre-trained model is used. We used stochastic gradient descent with a batch size of 64 and a learning rate of 0.01, decaying it by a factor of 0.3 once the loss plateaued. Dataset splits and batch shuffle are identically in all UCB experiments and all baselines. Pruning procedure and mask size: Once a task is learned, we compute the performance drop for a set of arbitrary pruning percentages from the maximum training accuracy achieved when no pruning is applied. The pruning portion is then chosen using a threshold beyond which the performance drop is not accepted. Mask size is chosen without having the knowledge of how many tasks to learn in the future. Upon learning each task we used a uniform distribution of pruning ratios (50-100%) and picked the ratio ed in at most 1%, 2%, and 3% forgetting for MNIST, CIFAR, and 8tasks experiments, respectively. We did not tune this parameter because in our hyperparameter tuning, we only assume we have validation sets of the first two tasks. Parameter regularization and importance measurement: Table 1 ablates different ways to compute the importance Ω of an parameter in Eq. 4 and 5. As shown in Table 1 the configuration that yields the highest accuracy and the least forgetting (maximum BWT) occurs when the learning rate regularization is performed only on µ of the posteriors using Ω µ = 1/σ as the importance and Ω ρ = 1. Performance measurement: Let n be the total number of tasks. Once all are learned, we evaluate our model on all n tasks. ACC is the average test classification accuracy across all tasks. To measure forgetting we report backward transfer, BWT, which indicates how much learning new tasks has influenced the performance on previous tasks. While BWT < 0 directly reports catastrophic forgetting, BWT > 0 indicates that learning new tasks has helped with the preceding tasks. Formally, BWT and ACC are as follows: where R i,n is the test classification accuracy on task i after sequentially finishing learning the n th task. Note that in UCB-P, R i,i refers the test accuracy on task i before pruning and R i,n after pruning which is equivalent to the end of sequence performance. In Section 6, we show that our UCB model can be used when tasks labels are not available at inference time by training it with a "single head" architecture with a sum of number of classes for all tasks. We refer to the ACC measured for this scenario as "Generalized Accuracy". We first present our for class incremental learning of MNIST (5-Split MNIST) in which we learn the digits 0 − 9 in five tasks with 2 classes at a time in 5 pairs of 0/1, 2/3, 4/5, 6/7, and 8/9. Table 2a shows the for reference baselines in Bayesian and non-Bayesian neural networks including fine-tuning (BBB-FT, ORD-FT), feature extraction (BBB-FE, ORD-FE) and, joint training (BBB-JT, ORD-JT) averaged over 3 runs and standard deviations are given in Table?? in the appendix. Although the MNIST dataset is an "easy" dataset, we observe throughout all experiments that Bayesian fine-tuning and joint training perform significantly better than their counterparts, ORD-FT and ORD-JT. For Bayesian methods, we compare against VCL and its variations named as VCL with Variational Adam (VCL-Vadam), VCL with Adam and Gaussian natural gradients (VCL-GNG). For non-Bayesian methods, we compare against HAT, IMM, and EWC (EWC can be regarded as Bayesian-inspired). VCL-Vadam (ACC=99.17%) appears to be outperforming VCL (ACC=98.20%) and VCL-GNG (ACC=96.50%) in average accuracy. However, full comparison is not possible because forgetting was not reported for Vadam and GNG. Nevertheless, UCB (ACC=99.63%) is able to surpass all the baselines including VCL-Vadam in average accuracy while in zero forgetting it is on par with HAT (ACC=99.59%). We also report on incrementally learning MNIST in two tasks (2-Split MNIST) in Table 8 in the appendix, where we compare it against PackNet, HAT, and LWF where PackNet, HAT, UCB-P, and UCB have zero forgetting while UCB has marginally higher accuracy than all others. Permuted MNIST is a popular variant of the MNIST dataset to evaluate continual learning approaches in which each task is considered as a random permutation of the original MNIST pixels. Following the literature, we learn a sequence of 10 random permutations and report average accuracy at the end. Table 2b shows ACC and BWT of UCB and UCB-P in comparison to state-of-the-art models using a small and a large network with 0.1M and 1.9M parameters, respectively (architecture details are given in Section A.2 of the appendix). The accuracy achieved by UCB (ACC=91.44 ± 0.04%) using the small network outperforms the ACC reported by for SI (ACC=86.0%), EWC (ACC=88.2%), while HAT attains a slightly better performance (ACC=91.6%). Comparing the average accuracy reported in VCL-Vadam (ACC=86.34%) and VCL-GNG (ACC=90.50%) as well as obtained for VCL (ACC=88.80%) shows UCB with BWT=(0.03% ± 0.00%) is able to outperform other Bayesian approaches in accuracy while forgetting significantly less compared to VCL with BWT=−7.9%. While we do not experiment with memory in this work, not surprisingly adding memory to most approaches will improve their performance significantly as it allows looking into past tasks. E.g. report ACC=94.37% for VCL-GNC when adding a memory of size 200. Next, we compare the for the larger network (1.9M). While HAT and UCB have zero forgetting, UCB, reaching ACC=97.42 ± 0.01%, performs better than all baselines including HAT which obtains ACC=97.34 ± 0.05% using 1.9M parameters. We also observe again that BBB-FT, despite being not specifically penalized to prevent forgetting, exhibits reasonable negative BWT values, performing better than IMM and LWF baselines. It is close to joint training, BBB-JT, with ACC=98.1%, which can be seen as an upper bound. In this experiment, we randomly alternate between class incremental learning of CIFAR10 and CIFAR100. Both datasets are divided into 5 tasks each with 2 and 20 classes per task, respectively. Table 2c presents ACC and BWT obtained with UCB-P, UCB, and three BBB reference methods compared against various continual learning baselines. Among the baselines presented in Table 2c, PNN and PathNet are the only zero-forgetting-guaranteed approaches. It is interesting to note that in this setup, some baselines (PathNet, LWF, and LFL) do not perform better than the naive accuracy achieved by feature extraction. PathNet suffers from bad pre-assignment of the network's capacity per task which causes poor performance on the initial task from which it never recovers. IMM performs almost similar to fine-tuning in ACC, yet forgets more. PNN, EWC, and HAT are the only baselines that perform better than BBB-FE and BBB-FT. EWC and HAT are both allowed to forget by construction, however, HAT shows zero forgetting behavior. While EWC is outperformed by both of our UCB variants, HAT exhibits 1% better ACC over UCB-P. Despite having a slightly higher forgetting, the overall accuracy of UCB is higher, reaching 79.4%. Finally, we present our for continual learning of 8 tasks using UCB-P and UCB in Table 2d. Similar to the previous experiments we look at both ACC and BWT obtained for UCB-P, UCB, BBB references (FT, FE, JT) as well as various baselines. Considering the ACC achieved by BBB-FE or BBB-FT (58.1%) as a lower bound we observe again that some baselines are not able to do better than BBB-FT including LFL, PathNet, LWF, IMM, and EWC while PNN and HAT remain the only strong baselines for our UCB-P and UCB approaches. UCB-P again outperforms PNN by 3.6% in ACC. HAT exhibits only −0.1% BWT, but our UCB achieves 2.4% higher ACC. 6 SINGLE HEAD AND GENERALIZED ACCURACY OF UCB UCB can be used even if the task information is not given at test time. For this purpose, at training time, instead of using a separate fully connected classification head for each task, we use a single head with the total number of outputs for all tasks. For example in the 8-dataset experiment we only use one head with 293 number of output classes, rather than using 8 separate heads, during training and inference time. for UCB from training with multi-head to a single head. The ACC reduction is 0.3%, 2.6%, 5.1%, and 4.1% for 2-Split MNIST, Permuted MNIST, Alternating CIFAR10/100, and sequence of 8 tasks experiments, respectively. We evaluated UCB and BBB-FT with a more challenging metric where the prediction space covers the classes across all the tasks. Hence, confusion of similar class labels across tasks can be measured. Performance for this condition is reported as Generalized ACC in Table 3 in columns 2-3. We observe a small performance reduction in going from ACC to Generalized ACC, suggesting non-significant confusion caused by the presence of more number of classes at test time. The performance degradation from ACC to Generalized ACC is 0.2%, 2.6%, 3.1%, and 3.1% for 2-Split MNIST, Permuted MNIST, Alternating CIFAR10/100, and sequence of 8 tasks, respectively. This shows that UCB can perform competitively in more realistic conditions such as unavailability of task information at test time. We believe the main insight of our approach is that instead of computing additional measurements of importance, which are often task, input or output dependent, we directly use predicted weight uncertainty to find important parameters. We can freeze them using a binary mask, as in UCB-P, or regularize changes conditioned on current uncertainty, as in UCB. In this work, we propose a continual learning formulation with Bayesian neural networks, called UCB, that uses uncertainty predictions to perform continual learning: important parameters can be either fully preserved through a saved binary mask (UCB-P) or allowed to change conditioned on their uncertainty for learning new tasks (UCB). We demonstrated how the probabilistic uncertainty distributions per weight are helpful to continually learning short and long sequences of benchmark datasets compared against baselines and prior work. We show that UCB performs superior or on par with state-of-the-art models such as HAT across all the experiments. Choosing between the two UCB variants depends on the application scenario: While UCB-P enforces no forgetting after the initial pruning stage by saving a small binary mask per task, UCB does not require additional memory and allows for more learning flexibility in the network by allowing small forgetting to occur. UCB can also be used in a single head setting where the right subset of classes belonging to the task is not known during inference leading to a competitive model that can be deployed where it is not possible to distinguish tasks in a continuous stream of the data at test time. UCB can also be deployed in a single head scenario and where tasks information is not available at test time. A APPENDIX A.1 DATASETS Table 4 shows a summary of the datasets utilized in our work along with their size and number of classes. In all the experiments we resized images to 32 × 32 × 3 if necessary. For datasets with monochromatic images, we replicate the image across all RGB channels. 10 60,000 10,000 CIFAR100 100 50,000 10,000 NotMNIST 10 16,853 1,873 SVHN 10 73,257 26,032 CIFAR10 10 39,209 12,630 TrafficSigns 43 39,209 12,630 FashionMNIST 10 60,000 10,000 In this section we take a closer look at elements of our UCB model on MNIST and evaluate variants of parameter regularization, importance measurement, as well as the effect of the number of samples drawn from the posited posterior. Bayes-by-backprop (BBB) Hyperparamters: Table 5 shows the search space for hyperparamters in the BBB algorithm which we used for tuning on the validation set of the first two tasks. Table 5: Search space for hyperparamters in BBB given by BBB hyperparamters − log σ 1 − log σ 2 π Search space {0, 1, 2} {6, 7, 8} {0.25,0.5,0.75} Network architecture: For Split MNIST and Permuted MNIST experiments, we have used a twolayer perceptron which has 1200 units. Because there is more number of parameters in our Bayesian neural network compared to its equivalent regular neural net, we ensured fair comparison by matching the total number of parameters between the two to be 1.9M unless otherwise is stated. For the multiple datasets learning scenario, as well as alternating incremental CIFAR10/100 datasets, we have used a ResNet18 Bayesian neural network with 7.1-11.3M parameters depending on the experiment. However, the majority of the baselines provided in this work are originally developed using some variants of AlexNet structure and altering that, e.g. to ResNet18, ed in degrading in their reported and experimented performance as shown in Table 6. Therefore, we kept the architecture for baselines as AlexNet and ours as ResNet18 and only matched their number of parameters to ensure having equal capacity across different approaches. Table 6: Continually learning on CIFAR10/100 using AlexNet and ResNet18 for UCB (our method) and HAT. BWT and ACC in %. All are (re)produced by us. Number of Monte Carlo samples: UCB is ensured to be robust to random noise using multiple samples drawn from posteriors. Here we explore different number of samples and the effect on final performance for ACC and BWT. We have used Ω µ = 1/σ as importance and regularization has been performed on mean values only. Following the in Table 7 we chose the number of samples to be 10 for all experiments. Here we include some additional such as Table 8 for 2-split MNIST and some complementary for tables in the main text as follows:??, 9, and 10 include standard deviation for shown in Table 2a, 2b, 2c, respectively.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

A regularization-based approach for continual learning using Bayesian neural networks to predict parameters' importance

We find that deep networks which generalize poorly are more reliant on single directions than those that generalize well, and evaluate the impact of dropout and batch normalization, as well as class selectivity on single direction reliance.