Concurrent Learning of Semantic Relations
Georgios Balikas Gaël Dias Rumen Moraliyski Massih-Reza Amini

Semi-regular Dubuc-Deslauriers wavelet tight frames
    Alberto Viscardi
    
===================================================




Discovering whether words are semantically related and identifying the specific semantic relation that holds between them is of crucial importance for NLP as it is essential for tasks like query expansion in IR. Within this context, different methodologies have been proposed that either exclusively focus on a single lexical relation (e.g. hypernymy vs. random) or learn specific classifiers capable of identifying multiple semantic relations (e.g. hypernymy vs. synonymy vs. random). In this paper, we propose another way to look at the problem that relies on the multi-task learning paradigm. In particular, we want to study whether the learning process of a given semantic relation (e.g. hypernymy) can be improved by the concurrent learning of another semantic relation (e.g. co-hyponymy). Within this context, we particularly examine the benefits of semi-supervised learning where the training of a prediction function is performed over few labeled data jointly with many unlabeled ones. Preliminary results based on simple learning strategies and state-of-the-art distributional feature representations show that concurrent learning can lead to improvements in a vast majority of tested situations.      








§ INTRODUCTION


The ability to automatically identify semantic relations is an important issue for Information Retrieval (IR) and Natural Language Processing (NLP) applications such as question answering <cit.>, query expansion <cit.>, or text summarization <cit.>. Semantic relations embody a large number of symmetric and asymmetric linguistic phenomena such as synonymy (bike ↔ bicycle), co-hyponymy (bike ↔ scooter), hypernymy (bike → tandem) or meronymy (bike → chain), but more can be enumerated <cit.>. 

Most approaches focus on modeling a single semantic relation and consist in deciding whether a given relation r holds between a pair of words (x,y) or not. Within this context, the vast majority of efforts <cit.> concentrate on hypernymy which is the key organization principle of semantic memory, but studies exist on antonymy <cit.>, meronymy <cit.> and co-hyponymy <cit.>. Another research direction consists in dealing with multiple semantic relations at a time and can be defined as deciding which semantic relation r_i (if any) holds between a pair of words (x,y). This multi-class problem is challenging as it is known that distinguishing between different semantic relations (e.g. synonymy and hypernymy) is difficult <cit.>. Within the CogALex-V shared task[<https://sites.google.com/site/cogalex2016/home/shared-task>] which aims at tackling synonymy, antonymy, hypernymy and meronymy as a multi-class problem, best performing systems are proposed by <cit.> and <cit.>.  

In this paper, we propose another way to look at the problem of semantic relation identification based on the following findings. First, symmetric similarity measures, which capture synonymy <cit.> have shown to play an important role in hypernymy detection <cit.>. Second, <cit.> show that learning term embeddings that take into account co-hyponymy similarity improves supervised hypernymy identification. As a consequence, we propose to study whether the learning process of a given semantic relation can be improved by the concurrent learning of another relation, where semantic relations are either synonymy, co-hyponymy or hypernymy. For that purpose, we propose a multi-task learning strategy using a hard parameter sharing neural network model that takes as input a learning word pair (x,y) encoded as a feature vector representing the concatenation[Best configuration reported in <cit.> for standard non path-based supervised learning.] of the respective word embeddings of x and y noted x⊕y. The intuition behind our experiments is that if the tasks are correlated, the neural network should improve its generalization ability by taking into account the shared information.

In parallel, we propose to study the generalization ability of the multi-task learning model based on a limited set of labeled word pairs and a large number of unlabeled samples, i.e. following a semi-supervised paradigm. As far as we know, most related works rely on the existence of a huge number of validated word pairs present in knowledge databases (e.g. WordNet) to perform the supervised learning process. However, such resources may not be available for specific languages or domains. Moreover, it is unlikely that human cognition and its generalization capacity rely on the equivalent number of positive examples. As such, semi-supervised learning proposes a much more interesting framework where unlabeled word pairs can massively[Even though with some error rate.] be obtained through selected lexico-syntactic patterns <cit.> or paraphrase alignments <cit.>. To test our hypotheses, we propose a self-learning strategy where confidently tagged unlabeled word pairs are iteratively added to the labeled dataset.  

Preliminary results based on simple (semi-supervised) multi-task learning models with state-of-the-art word pairs representations (i.e. concatenation of GloVe <cit.> word embeddings) over the gold standard dataset ROOT9 <cit.> and the RUMEN dataset proposed in this paper show that classification improvements can be obtained for a wide range of tested configurations.      








§ RELATED WORK


Whether semantic relation identification has been tackled as a one-class or a multi-class problem, two main approaches have been addressed to capture the semantic links between two words (x,y): pattern-based and distributional. Pattern-based (a.k.a. path-based) methods base their decisions on the analysis of the lexico-syntactic patterns (e.g. X such as Y) that connect the joint occurrences of x and y. Within this context, earlier works have been proposed by <cit.> (unsupervised) and <cit.> (supervised) to detect hypernymy. However, this approach suffers from sparse coverage and benefits precision over recall. To overcome these limitations, recent one-class studies on hypernymy <cit.> and antonymy <cit.>, as well as multi-class approaches <cit.> have been focusing on representing dependency patterns as continuous vectors using long short-term memory (LSTM) networks. Within this context, successful results have been evidenced but <cit.> also show that the combination of pattern-based methods with the distributional approach greatly improves performance. 

In distributional methods, the decision whether x is within a semantic relation with y is based on the distributional representation of these words following the distributional hypothesis <cit.>, i.e. on the separate contexts of x and y. Earlier works developed symmetric <cit.> and asymmetric <cit.> similarity measures based on discrete representation vectors, followed by numerous supervised learning strategies for a wide range of semantic relations <cit.>, where word pairs are encoded as the concatenation of the constituent words representations (x⊕y) or their vector difference (x - y). More recently, attention has been focusing on identifying semantic relations using neural language embeddings, as such semantic spaces encode linguistic regularities <cit.>. Within this context, <cit.> proposed an exhaustive study for a wide range of semantic relations and showed that under suitable supervised training, high performance can be obtained. However, <cit.> also showed that some relations such as hypernymy are more difficult to model than others. As a consequence, new proposals have appeared that tune word embeddings for this specific task, where hypernyms and hyponyms should be closed to each other in the semantic space <cit.>.

In this paper, we propose an attempt to deal with semantic relation identification based on a multi-task strategy, as opposed to previous one-class and multi-class approaches. Our main scope is to analyze whether a link exists between the learning process of related semantic relations. The closest approach to ours is proposed by <cit.>, which develops a multi-task convolutional neural network for multi-class semantic relation classification supported by relatedness classification. As such, it can be seen as a domain adaptation problem. Within the scope of our paper, we aim at studying semantic inter-relationships at a much finer grain and understanding the cognitive links that may exist between synonymy, co-hyponymy and hypernymy, that form the backbone of any taxonomic structure as illustrated in Figure <ref>. For this first attempt, we follow the distributional approach as in <cit.>, although we are aware that improvements may be obtained by the inclusion of pattern-based representations[This issue is out of the scope of this paper.]. Moreover, we propose the first attempt[As far as we know.] to deal with semantic relation identification based on a semi-supervised approach, thus avoiding the pre-existence of a large number of training examples. As a consequence, we aim at providing a more natural learning framework where only a few labeled examples are initially provided and massively gathered related word pairs iteratively improve learning.










§ METHODOLOGY











 §.§ Multi-task with hard parameter sharing

As discussed in <cit.>, not every task combination is beneficial. But, concurrent learning of tasks that have cognitive similarities is often beneficial. We may hypothesize that recognizing the different semantic relations that hold between words can benefit classification models across similar tasks. For instance, learning that bike is the hypernym of mountain bike should help while classifying mountain bike and tandem bicycle as co-hyponyms, as it is likely that tandem bicycle shares some relation with bike. To test this hypothesis, we propose to use a multi-task learning approach. Multi-task learning <cit.> has empirically been validated and has shown to be effective in a variety of NLP tasks ranging from sentiment analysis, part-of-speech tagging and text parsing <cit.>. The hope is that by jointly learning the decision functions for related tasks, one can achieve better performance. It may be first due to knowledge transfer across tasks that is achieved either in the form of learning more robust representations or due to the use of more data. Second, it has been argued that multi-task learning can act as a regularization process thus preventing from overfitting by requiring competitive performance across different tasks <cit.>. 

In this paper, we propose to use a multi-task learning algorithm that relies on hard parameter sharing[We use a simple architecture as our primary objective is to validate our initial hypotheses and not necessarily focus on overall performance.]. The idea is that the shared parameters (e.g. word representations or weights of some hidden layers) can benefit the performance of all tasks learned concurrently if the tasks are related. In particular, we propose a hard parameter sharing architecture based on a feed-forward neural network (NN) to perform the classification step. The NN architecture is illustrated in Figure <ref>. 




The input of the network is the concatenation of the word embeddings of the word pairs followed by a series of non-linear hidden layers. Then, a number of softmax layers gives the network predictions. Here, a softmax layer corresponds to a task, and concurrently learning M tasks requires M separate output softmax layers. The efficiency of hard parameter sharing architectures relies on the fact that the first layers that are shared are tuned by back-propagating the classification errors of every task. That way, the architecture uses the datasets of all tasks, instead of just one at a time. In Algorithm <ref>, we detail the training protocol. Note that the different tasks learned by the NN share the same weights as batches are randomly sampled from their corresponding datasets[Automatically learning different weights for the tasks and self-adjusting them for the sake of overall performance is a possible future research direction.]. 





 §.§ Semi-supervision via self-learning

Semi-supervised learning approaches have shown to perform well in a variety of tasks such as text classification and text summarization <cit.>. As in the supervised learning framework, we assume that we are given access to a set ℒ={(w_i,w_i',rel)}_i=1^i=K that consists of K pairs of words labeled according to the relationship rel. Complementary to that, we also assume to have access to a set of K' words pairs 𝒰={(w_i,w_i')}_i=1^i=K' distinct from those of ℒ, and totally unlabeled. The challenge in this setting is to surpass the performance of classification models trained exclusively on ℒ by using the available data in 𝒰. While several methods have been proposed, we opt to use self-learning as the semi-supervised algorithm, as it is one of the simplest approaches[The main objective of this paper is to verify the benefits of semi-supervised multi-task learning for our task and not to tune complex solutions that can reach high performance. This remains a future work.]. The central idea behind self-learning is to train a learner on the set ℒ, and then progressively expand ℒ, by pseudo-labeling N pairs within 𝒰, for which the current prediction function is the most confident and adding them to ℒ. This process is repeated until no more pairs are available in 𝒰 or, that the performance on a validation set degrades due to the newly-added possibly noisy examples. Algorithm <ref> details this process. 



One point illustrated in Algorithm <ref> to be highlighted  is that the training set ℒ is augmented after each iteration of self-learning in a stratified way. In this case, the class distribution of the N pseudo-labeled examples that are added to ℒ is the same as the class distribution of ℒ. This constraint follows from the independent and identically distributed (i.i.d.) assumption between the ℒ and 𝒰 sets and ensures that the distribution on the classes in the training set does not change as training proceeds. Another point to be mentioned is that the examples that are added to ℒ may be noisy. Despite the confident predictions of the classifier C, one should expect that some of the instances added are wrongly classified. To reduce the impact of the noise to the training set, we monitor the performance of the classifier using the validation set 𝒱 and if the performance degrades the self-learning iteration stops.







§ EXPERIMENTAL SETUPS



In this section, we present the experimental setups of our study so that our results can easily be reproduced. In particular, we detail the learning frameworks, and the datasets and their splits.



 §.§ Learning Frameworks

In order to evaluate the effects of our learning strategy, we implement the following baseline systems: (1) Majority Baseline; (2) Logistic Regression that has shown positive results in <cit.>, and (3) Feed-forward neural network with two hidden layers of 50 neurons each, which is the direct one-task counterpart of our NN multi-task architecture. For the multi-task learning algorithm, we implemented the architecture shown in Figure <ref> using Keras <cit.>. In particular, we define 2 fully- connected hidden layers  (i.e. h_1, h_2, R=2) of 50 neurons each. While the number of hidden layers is a free parameter to tune, we select two hidden layers in advance so that the complexity of the multi-task models are comparable to the neural network baseline. The word embeddings are initialized with the 300-dimensional representations of GloVe <cit.>. The activation function of the hidden layers is the sigmoid function and the weights of the layers are initialized with a uniform distribution scaled as described in <cit.>. As for the learning process, we use the Root Mean Square Propagation (RMSprop) optimization method with learning rate set to 0.001 and the default value for ρ=0.9. 
For every task, we use the binary cross-entropy loss function. The network is trained with batches of 32 examples[The code is freely available upon demand for research purposes.]. For the Logistic Regression, we used the implementation of scikit-learn <cit.>.











 §.§ Datasets and Splits




In order to perform our experiments, we use the ROOT9 dataset[https://github.com/esantus/ROOT9] <cit.> that contains 9,600 word pairs, randomly extracted from three well-known datasets: EVALution <cit.>, Lenci/Benotto <cit.> and BLESS <cit.>. The word pairs are equally distributed among three classes (hypernymy, co-hyponymy and random) and involve several part-of-speech tags (adjectives, nouns and verbs). Here, we exclusively focus on nouns and keep 1,212 hypernyms, 1,604 co-hyponyms and 549 random pairs that can be represented by GloVe  embeddings <cit.>.

In order to include synonymy as a third studied semantic relation, we build the RUMEN dataset[The dataset is freely available upon demand for research purposes.] that contains 18,978 word pairs equally organized amongst three classes (hypernymy, synonymy and random). By doing so, we expect to move to more challenging settings and overcome the limitations of only using hypernyms and co-hyponyms  <cit.>. In RUMEN, all noun pairs are randomly selected based on WordNet 3.0[http://wordnetcode.princeton.edu/3.0/] <cit.> such that hypernyms are not necessarily in direct relation and random pairs have as most common parent the root of the hierarchy with a minimum path distance equals to 7[This value was set experimentally.] to ensure semantic separateness. Finally, we keep 3,375 hypernym, 3,213 synonym and 3,192 random word pairs encoded by GloVe embeddings.

Following a classical learning procedure, the datasets must be split into different subsets: train, validation, test and unlabeled in the case of semi-supervision. The standard procedure is random splitting where word pairs are randomly selected without other constraint to form the subsets. However, <cit.> point out that using distributional representations in the context of supervised learning tends to perform lexical memorization. In this case, the model mostly learns independent properties of single terms in pairs. For instance, if the training set contains word pairs like (bike, tandem), (bike, off-roader) and (bike, velocipede) tagged as hypernyms, the algorithm may learn that bike is a prototypical hypernym and all new pairs (bike, y) may be classified as hypernyms, regardless of the relation that holds between bike and y. To overcome this situation and prevent the model from overfitting by lexical memorization, <cit.> suggested to split the train and test sets such that each one contains a distinct vocabulary. This procedure is called lexical split. Within the scope of this study, we propose to apply lexical split as defined in <cit.>. So, lexical repetition exists in the train, validation and the unlabeled subsets, but the test set is  exclusive in terms of vocabulary. Table <ref> shows the vocabulary and the pairs before and after the lexical splits. In our experiments, we have further split the pairs dubbed as train so that 60% of them are unlabeled examples. From the remaining 40%, we have randomly selected 30% for validation, resulting in few training examples, which resembles more to a realistic learning scenario where only few positive examples are known. So, while lexical split ensures that the network generalizes to unseen words, it also results in significantly smaller datasets due to the way that these datasets are produced. All subsets are available for replicability[All datasets are freely available upon demand for research purposes.].









§ RESULTS


In the experiments that follow, we report two evaluation measures: Accuracy and Macro-average F_1 measure (MaF_1). Accuracy captures the number of correct predictions over the total predictions, while MaF_1 evaluates how the model performs across the different relations as it averages the F_1 measures of each relation without weighting the number of examples in each case. In the remaining of this section, we comment on three experiments.




In the first experiment, we propose to study the impact of the concurrent learning of co-hyponymy (bike ↔ scooter) and hypernymy (bike → tandem) following the first findings of <cit.>. For that purpose, we propose to apply our (semi-supervised) multi-task learning strategy over the lexically split ROOT9 dataset using vector concatenation of GloVe <cit.> as feature representation. Results are illustrated in Table <ref>. The multi-task paradigm shows that an improved MaF_1 score can be achieved by concurrent learning without semi-supervision achieving a value of 77.3% (maximum value overall). In this case, a 1.1% improvement is obtained over the best baseline (i.e. logistic regression) for hypernymy classification, indeed suggesting that there exists a learning link between hypernymy and co-hyponymy. However, the results for co-hyponymy classification can not compete with a classical supervised strategy using logistic regression. In this case, a 2.1% decrease in MaF_1 is evidenced suggesting that the gains for hypernymy classification are not positively balanced by the performance of co-hyponymy. So, we can expect an improvement for hypernymy classification but not for co-hyponymy in a multi-task environment, suggesting a positive influence of co-hyponymy learning towards hypernymy but not the opposite. Interestingly, the results of the semi-supervised strategy reach comparable figures compared to the multi-task proposal (even superior in some cases), but do not complement each other for the semi-supervised multi-task experiment. In this case, worst results are obtained for both classification tasks suggesting that the multi-task model is not able to correctly generalize from a large number of unlabeled examples, while this is the case for the one-task architecture. 

In the second experiment, we propose to study the impact of the concurrent learning of synonymy (bike ↔ bicycle) and hypernymy following the experiments of <cit.> which suggest that symmetric similarity measures (usually tuned to detect synonymy <cit.>) improve hypernymy classification. For that purpose, we propose to apply the same models over the lexically split RUMEN dataset. Results are illustrated in Table <ref>. The best configuration is the combination of multi-task learning with self-learning achieving maximum accuracy and MaF_1 scores for both tasks. The improvement equals to 0.7% in terms of MaF_1 for hypernymy and reaches 3% in terms of MaF_1 for synonymy when compared to the best baseline (i.e. neural network). The overall average improvement (i.e. both tasks combined[Column 3 of Table <ref>.]) reaches 1.8% for accuracy and 1.9% for MaF_1 over the best baseline. So, these results tend to suggest that synonymy identification may greatly be impacted by the concurrent learning of hypernymy and vice versa (although to a less extent). In fact, these results consistently build upon the positive results of the multi-task strategy without semi-supervision and the self-learning approach alone that both improve over the best baseline results. Note that the results obtained over the RUMEN dataset by the baseline classifiers are lower than the ones reached over ROOT9 for hypernymy, certainly due to the complexity of the datasets themselves. So, we may hypothesize that the multi-task strategy plays an important role by acting as a regularization process and helping in solving learning ambiguities, and reaches improved results over the one-task classifiers. 

In the third experiment, we propose to study the impact of the concurrent learning of co-hyponymy, synonymy and hypernymy all together. The idea is to understand the inter-relation between these three semantic relations that form the backbone of any taxonomic structure. For that purpose, we propose to apply the models proposed in this paper over the lexically split ROOT9+RUMEN dataset[Note that due to the lexical split process, results can not directly be compared to the ones obtained over ROOT9 or RUMEN.]. Results are illustrated in Table <ref>. The best configuration for all the tasks combined (i.e. co-hyponymy, synonymy and hypernymy) is multi-task learning without semi-supervision. Overall, improvements up to 1.4% in terms of accuracy and 2% in terms of MaF_1 can be reached over the best baseline (i.e. neural network). In particular, the MaF_1 score increases 4.4% with the multi-task strategy without self-learning for co-hyponymy, while the best result for synonymy is obtained by the semi-supervised multi-task strategy with an improvement of 1.1% MaF_1 score. The best configuration for hypernymy is evidenced by self-learning alone, closely followed by the multi-task model, reaching improvements in MaF_1 scores of 1.7% (resp. 1%) for self-learning (resp. multi-task learning). Comparatively to the first experiment, both learning paradigms (i.e. semi-supervision and multi-task) tend to produce competitive results alone, both exceeding results of the best baseline. However, the multi-task model hardly generalizes from the set of unlabeled examples, being synonymy the only exception. Finally, note that co-hyponymy seems to be the simplest task to solve, while synonymy is the most difficult one, over all experiments.   





§ STUDYING MERONYMY




In this section, we study the introduction of the meronymy relation (bike → chain) into a multi-task environment, as it has traditionally been studied together with hypernymy <cit.>. The overall idea is to verify whether the meronymy semantic relation can benefit from the concurrent learning of the backbone semantic relations that form knowledge bases. For that purpose, we apply our learning models over the lexically split BLESS dataset <cit.> that includes three semantic relations: co-hyponymy, hypernymy and meronymy. The details of the lexical split is presented in Table <ref> and note that the BLESS dataset has been processed in the exact same way as ROOT9 and RUMEN, i.e. retaining only noun categories and word pairs that can be represented by the GloVe semantic space. Results are presented in Table <ref>. The best configuration over the three tasks combined is obtained by the semi-supervised multi-task strategy with a MaF_1 score equals to 80.3%, thus improving 1.2% over the best baseline (i.e. neural network). In particular, we can notice that the most important improvement is obtained for the meronymy relation that reaches 73.3% for MaF_1 and 76.4% for accuracy with the multi-task model without semi-supervision. In this particular case, the improvement is up to 2.6% in accuracy and 2.4% in MaF_1 over the neural network baseline. For co-hyponymy (resp. hypernymy), best results are obtained by multi-task with semi-supervision (resp. without semi-supervision), but show limited improvements over the best baseline, suggesting that meronymy gains more in performance from the concurrent learning of co-hyponymy and hypernymy than the contrary, although improvements are obtained in all cases. Comparatively to the other experiments, we also notice that although the self-learning algorithm and the multi-task framework without semi-supervision perform well alone, the combination of both strategies does not necessary lead to the best results overall, suggesting that the present architecture can be improved to positively gain from the massive extraction of unlabeled examples.    






§ CONCLUSIONS AND DISCUSSION


In this paper, we proposed to study the concurrent learning of semantic relations (co-hyponymy, hypernymy, synonymy and meronymy) using simple learning strategies (self-learning and hard parameter sharing multi-task learning) and state-of-the-art continuous distributional representations (concatenation of GloVe embeddings). The idea was to verify if the concurrent learning of these relations that share some cognitive similarities could be beneficial, without necessarily focusing on overall performance. Obtained results show that concurrent learning can lead to improvements in a vast majority of tested situations. In particular, we have shown that within this experimental framework, hypernymy can gain from co-hyponymy, synonymy from hypernymy, co-hyponymy from both hypernymy and synonymy, and meronymy from both co-hyponymy and hypernymy. Moreover, it is interesting to notice that in three cases out of four, the improvement obtained by the multi-task strategy is obtained for the most difficult task to handle, thus suggesting the benefits of concurrent learning. Based on these preliminary findings, a vast amount of improvements can now be introduced into the framework to increase overall performance. With respect to the input features, we intend to study the potential benefits from dedicated embeddings such as hypervec <cit.>, knowledge graphs embeddings <cit.> and dual embeddings <cit.>. Moreover, we deeply believe that the LSTM path-based features introduced in <cit.> and some well-defined word pairs similarity measures <cit.> can lead to classification improvements by complementing the information present in distributional semantic spaces. With respect to the learning framework, some clear efforts must be performed. Indeed, the current combination of self-learning and hard parameter sharing multi-task learning is not beneficial in a vast majority of cases suggesting the proposition of new architectures following the ideas of Tri-training <cit.>. Moreover, it is clear that more complex architectures such as convolutional neural networks may improve the learning process as it is proposed in <cit.> for similar tasks. As semi-supervision is concerned, we intend to study a more realistic situation where unlabeled examples are massively gathered by lexico-syntactic patterns <cit.> or by paraphrase alignment <cit.> over huge raw text corpora. Indeed, here, semi-supervision is just simulated by fractioning the existing datasets. Finally, we plan to difficult the original task by including the detection of the direction of the asymmetric semantic relations, testing the combination of more closely related semantic relations and including noisy pairs as in <cit.>. 























acl_natbib_nourl


