Still a Pain in the Neck: Evaluating Text Representations on Lexical Composition
Vered Shwartz Ido Dagan

[
    N. Korolkova^1 and G. Leuchs^2
    
==================================


Building meaningful phrase representations is challenging because phrase meanings are not simply the sum of their constituent meanings. Lexical composition can shift the meanings of the constituent words and introduce implicit information. We tested a broad range of textual representations for their capacity to address these issues. We found that, as expected, contextualized word representations perform better than static word embeddings, more so on detecting meaning shift than in recovering implicit information, in which their performance is still far from that of humans. Our evaluation suite, consisting of six tasks related to lexical composition effects, can serve future research aiming to improve representations.  




§ INTRODUCTION


Modeling the meaning of phrases involves addressing semantic phenomena that pose non-trivial challenges for common text representations, which derive a phrase representation from those of its constituent words. One such phenomenon is meaning shift, which happens when the meaning of the phrase departs from the meanings of its constituent words. This is especially common among verb-particle constructions (carry on), idiomatic noun compounds (guilt trip) and other multi-word expressions (MWE, lexical units that form a distinct concept), making them “a pain in the neck” for NLP applications <cit.>. 

A second phenomenon is common for both MWEs and free word combinations such as noun compounds and adjective-noun compositions. It happens when the composition introduces an implicit meaning that often requires world knowledge to uncover. For example, that hot refers to the temperature of tea but to the manner of debate <cit.>, or that olive oil is made of olives while baby oil is made for babies <cit.>. 

There has been a line of attempts to learn compositional phrase representations <cit.>, but many of these are tailored to a specific type of phrase or to a fixed number of constituent words, and they all disregard the surrounding context. Recently, contextualized word representations boasted dramatic performance improvements on a range of NLP tasks <cit.>. Such models serve as a function for computing word representations in a given context, making them potentially more capable to address meaning shift. These models were shown to capture some world knowledge <cit.>, which may potentially help with uncovering implicit information. 

In this paper we test how well various text representations address these composition-related phenomena. Methodologically, we follow recent work that applied “black-box” testing to assess various capacities of distributed representations <cit.>. We construct an evaluation suite with six tasks related to the above two phenomena, as shown in Figure <ref>, and develop generic models that rely on pre-trained representations. We test six representations, including static word embeddings <cit.> and contextualized word embeddings <cit.>. Our contributions are as follows:


    
  * We created a unified framework that tests the capacity of representations to address lexical composition via classification tasks, focusing on detecting meaning shift and recovering implicit meaning. We test six representations and provide an in depth analysis of the results.
    
  * We relied on existing annotated datasets used in various tasks, and re-casted them to fit our classification framework. We additionally annotated a sample from each test set to confirm the data validity and estimate the human performance on each task. 
    
  * We provide the classification framework, including data and code and available at <https://github.com/vered1986/lexcomp>, which would allow testing future models for their capacity to address lexical composition. 


Our results confirm that the contextualized word embeddings perform better than the static ones. In particular, we show that indeed modeling context contributes to recognizing meaning shift: on such tasks, contextualized models performed on par with humans. 

Conversely, despite hopes of filling missing information with world knowledge provided by the contextualized representations, the signal they yield for recovering implicit information is much weaker, and the gap between the best performing model and the human performance on such tasks remains substantial. We expect that improving the ability of such representations to reveal implicit meaning would require more than a language model training objective. In particular, one future direction is a richer training objective that simultaneously models multiple co-occurrences of the constituent words across different texts, as is commonly done in noun compound interpretation <cit.>. 









§ COMPOSITION TASKS


We experimented with six tasks that address the meaning shift and implicit meaning phenomena, summarized in Table <ref> and detailed below. 

We rely on existing tasks and datasets, but make substantial changes and augmentations in order to create a uniform framework. First, we cast all tasks as classification tasks. Second, we add sentential contexts where the original datasets annotate the phrases out-of-context, by extracting averaged-length sentences (15-20 words) from English Wikipedia (January 2018 dump) in which the target phrase appears. We assume that the annotation does not depend on the context, an assumption that holds in most cases, judging by the human performance scores (Section <ref>). Finally, we split each dataset to roughly 80% train and 10% for each of the validation and test sets, under lexical constraints as detailed for each task. 



 §.§ Recognizing Verb-Particle Constructions
  


A verb-particle construction (VPC) consists of a head verb and a particle, typically in the form of an intransitive preposition, which changes the verb's meaning (e.g. carry on vs. carry). 



  
Task Definition. Given a sentence s that includes a verb V followed by a preposition P, the goal is to determine whether it is a VPC or not. 



  
Data. We use the dataset of tu-roth:2012:STARSEM-SEMEVAL which consists of 1,348 sentences from the British National Corpus (BNC), each containing a verb V and a preposition P annotated to whether it is a VPC or not. The dataset is focused on 23 different phrasal verbs derived from six of the most frequently used verbs (take, make, have, get, do, give), and their combination with common prepositions or particles. To reduce label bias, we split the dataset lexically by verb, i.e. the train, test, and validation sets contain distinct verbs in their V and P combinations. 





 §.§ Recognizing Light Verb Constructions
  


The meaning of a light verb construction (LVC, e.g. make a decision) is mainly derived from its noun object (decision), while the meaning of its main verb (make) is “light” <cit.>. As a rule of thumb, an LVC can be replaced by the verb usage of its direct object noun (decide) without changing the meaning of the sentence.



  
Task Definition. Given a sentence s that includes a potential light verb construction (“make an easy decision”), the goal is to determine whether it is an LVC or not. 



  
Data. We use the dataset of tu-roth:2011:MWE, which contains 2,162 sentences from BNC in which a potential light verb construction was found (with the same 6 common verbs as in Section <ref>), annotated to whether it is an LVC in a given context or not. We split the dataset lexically by the verb. 



 §.§ Noun Compound Literality





  
Task Definition. Given a noun compound NC = w_1 w_2 in a sentence s, and a target word w ∈{w_1, w_2}, the goal is to determine whether the meaning of w in NC is literal. For instance, market has a literal meaning in flea market but flea does not. 



  
Data. We use the dataset of reddy-mccarthy-manandhar:2011:IJCNLP-2011 which consists of 90 noun compounds along with human judgments about the literality of each constituent word. Scores are given in a scale of 0-5, 0 being non-literal and 5 being literal. For each noun compound and each of its constituents we consider examples with a score ≥ 4 as literal, and ≤ 2 as non-literal, ignoring the middle range. We obtain 72 literal and 72 non-literal examples.

To increase the dataset size we augment it with literal examples from the tratz2011semantically dataset of noun compound classification. Compounds in this dataset are annotated to the semantic relation that holds between w_1 and w_2. Most relations (except for , which we ignore), define the meaning of NC as some trivial combination of w_1 and w_2, allowing us to regard both words as literal. This method produces additional 3,061 literal examples.



  
Task Adaptation. We add sentential contexts from Wikipedia, keeping up to 10 sentences per example. We downsample from the literal examples to balance the dataset, allowing for a ratio of up to 4 literal to non-literal examples. We split the dataset lexically by head, i.e. if w_1 w_2 is in one set, there are no w'_1 w_2 NCs in the other sets.[We chose to split by head rather than by modifier based on the majority baseline that achieved better performance.]




 §.§ Noun Compound Relations





  
Task Definition. Given a noun compound NC = w_1 w_2 in a sentential context s, and a paraphrase p, the goal is to determine whether p describes the semantic relation between w_1 and w_2 or not. For example, “part that makes up body” is a valid paraphrase for body part, but “replacement part bought for body” is not.



  
Data. We use the data from SemEval 2013 Task 4 <cit.>. The dataset consists of 356 noun compounds annotated with 12,446 human-proposed free text paraphrases.  



  
Task Adaptation. The goal of the SemEval task was to generate a list of free-form paraphrases for a given noun compound, which explicate the implicit semantic relation between its constituent. To match with our other tasks, we cast the task as a binary classification problem where the input is a noun compound NC and a paraphrase p, and the goal is to predict whether p is a correct description of NC. 

The positive examples for an NC w_1 w_2 are trivially derived from the original data by sampling up to 5 of its paraphrases and creating a (NC, p, TRUE) example for each paraphrase p. The same number of negative examples is then created using negative sampling from the paraphrase templates of other noun compounds w'_1 w_2 and w_1 w'_2 in the dataset that share a constituent word with NC. For example, “replacement part bought for body” is a negative example constructed from the paraphrase template “replacement [w_2] bought for [w_1]” which appeared for car part. We require one shared constituent in order to form more fluent paraphrases (which would otherwise be easily classifiable as negative). To reduce the chances of creating negative examples which are in fact valid paraphrases, we only consider negative paraphrases whose verbs never occurred in the positive paraphrase set for the given NC. 

We add sentential contexts from Wikipedia, randomly selecting one sentence per example, and split the dataset lexically by head. 





 §.§ Adjective-Noun Attributes
  




  
Task Definition. Given an adjective-noun composition AN in a sentence s, and an attribute AT, the goal is to determine whether AT is implicitly conveyed in AN. For example, the attribute  is conveyed in hot water, but not in hot argument ().



  
Data. We use the HeiPLAS data set <cit.>, which contains 1,598 adjective-noun compositions annotated with their implicit attribute meaning. The data was extracted from WordNet and manually filtered. The label set consists of attribute synsets in WordNet which are linked to adjective synsets.



  
Task Adaptation. Since the dataset is small and the number of labels is large (254), we recast the task as a binary classification task. The input to the task is an AN and a paraphrase created from the template “[A] refers to the [AT] of [N]” (e.g. “loud refers to the volume of thunder”). The goal is to predict whether the paraphrase is correct or not with respect to the given AN. 


We create up to 3 negative instances for each positive instance by replacing AT in another attribute that appeared with either A or N. For example, (hot argument, , False). To reduce the chance that the negative attribute is in fact a valid attribute for AN, we compute the Wu-Palmer similarity <cit.> between the original and negative attribute, taking only attributes whose similarity to the original attribute is below a threshold (0.4). 



Similarly to the previous task, we attach a context sentence from Wikipedia to each example. Finally, we split the dataset lexically by adjective, i.e. if A N is in one set, there are no A N' examples in the other sets. 



 §.§ Identifying Phrase Type



The last task consists of multiple phrase types and addresses detecting both meaning shift and implicit meaning.



  
Task Definition. The task is defined as sequence labeling to  tags. Given a sentence, each word is classified to whether it is part of a phrase, and the specific type of the phrase.



  
Data. We use the STREUSLE corpus <cit.>. The corpus contains texts from the web reviews portion of the English Web Treebank, along with various semantic annotations, from which we use the  annotations. Each token is labeled with a tag, a  tag marks the beginning of a span of type X,  occurs inside a span, and  outside of it.  labels mark specific types of phrase.[Sorted by frequency: noun phrase, weak (compositional) MWE, verb-particle construction, verbal idioms, prepositional phrase, auxiliary, adposition, discourse / pragmatic expression, inherently adpositional verb, adjective, determiner, adverb, light verb construction, non-possessive pronoun, full verb or copula, conjunction.]



  
Task Adaptation. We are interested in a simpler version of the annotations. Specifically, we exclude the discontinuous spans (e.g. a span like “turn the [TV] off” would not be considered as part of a phrase). The corpus distinguishes between “strong” MWEs (fixed or idiomatic phrases) and “weak” MWEs (ad hoc compositional phrases). The weak MWEs are untyped, hence we label them as  (compositional). 



§ REPRESENTATIONS


We experimented with 6 common word representations from two different families detailed below. Table <ref> summarizes the differences between the pretrained models used in this paper.



  
Word Embeddings. Word embedding models provide a fixed d-dimensional vector for each word in the vocabulary. Their training is based on the distributional hypothesis, according to which semantically-similar words tend to appear in the same contexts <cit.>. word2vec <cit.> can be trained with one of two objectives. We use the embeddings trained with the Skip-Gram objective which predicts the context words given the target word. GloVe <cit.> learns word vectors with the objective of estimating the log-probability of a word pair co-occurrence. fastText <cit.> extends word2vec by adding information about subwords (bag of character n-grams). This is especially helpful in morphologically-rich languages, but can also help handling rare or misspelled words in English. 



  
Contextualized Word Embeddings are functions computing dynamic word embeddings for words given their context sentence, largely addressing polysemy. They are pre-trained as general purpose language models using a large-scale unannotated corpus, and can later be used as a representation layer in downstream tasks (either fine-tuned to the task with the other model parameters or fixed). The representations used in this paper have multiple output layers. We either use only the last layer, which was shown to capture semantic information <cit.>, or learn a task-specific scalar mix of the layers (see Section <ref>). 

ELMo <cit.> are obtained by learning a character-based language model using a deep biLSTM <cit.>. Working at the character-level allows using morphological clues to form robust representations for out-of-vocabulary words, unseen in training. The OpenAI GPT <cit.> has a similar training objective, but the underlying encoder is a transformer <cit.>. It uses subwords as the basic unit, employing bytepair encoding. Finally, BERT <cit.> is also based on the transformer, but it is bidirectional as opposed to left-to-right as in the OpenAI GPT, and the directions are dependent as opposed to ELMo's independently trained left-to-right and right-to-left LSTMs. It also introduces a somewhat different objective called “masked language model”: during training, some tokens are randomly masked, and the objective is to restore them from the context. 



§ CLASSIFICATION MODELS


We implemented minimal “Embed-Encode-Predict” models that use the representations from Section <ref> as inputs, keeping them fixed during training. The rationale behind the model design was to keep them uniform for easy comparison between the representations, and make them simple so that the model's success can be attributed directly to the input representations. 



  
Embed. We use the embedding model to embed each word in the sentence s = w_1 ... w_n, obtaining:


    v⃗_1, ..., v⃗_n = Embed(s)


where v⃗_i stands for the word embedding of word w_i, which may be computed as a function of the entire sentence (in the case of contextualized word embeddings). 

Depending on the specific task, we may have another input w'_1, ..., w'_l to embed separately from the sentence: the paraphrases in the NC Relations and AN Attributes tasks, and the target word in the NC Literality task (to obtain an out-of-context representation of the target word). We embed this extra input as follows:


    v⃗'⃗_1, ..., v⃗'⃗_l = Embed(w'_1, ..., w'_l)






  
Encode. We encode the embedded sequences v⃗_1, ..., v⃗_n and v⃗'⃗_1, ..., v⃗'⃗_l using one of the following 3 encode variants. As opposed to the pre-trained embeddings, the encoder parameters are updated during training to fit the specific task.


    
  * : Encoding the embedded sequence using a biLSTM with a hidden dimension d, where d is the input embedding dimension: 

    
    u⃗_1, ..., u⃗_n = biLSTM(v⃗_1, ..., v⃗_n)

    
    
  * : Encoding the embedded sequence using self-attention. Each word is represented as the concatenation of its embedding and a weighted average over other words in the sentence:
    
    
    u⃗_i = [v⃗_i ; ∑_j=1^n a_i, j·v⃗_j ]

    
    The weights a_i are computed by applying dot-product between v⃗_i and every other word, and normalizing the scores using softmax:
    
    
    a⃗_i = softmax(v⃗_i^T ·v⃗)

    
    
  * : In which we don't encode the embedded text, but simply define:

    
    u⃗_1, ..., u⃗_n = v⃗_1, ..., v⃗_n

    


For all encoder variants, u⃗_i stands for the vector representing w_i, which is used as input to the classifier. 



  
Predict. We represent a span by concatenating its end-point vectors, e.g. u⃗_i...i+k = [u⃗_i ; u⃗_i+k] is the target span vector of w_i, ..., w_i+k. In tasks which require a second span, we similarly compute u⃗'⃗_1...l, representing the encoded span w'_1, ..., w'_l (e.g. the paraphrase in NC relations). The input to the classifier is the concatenation of u⃗_i...i+k, and, when applicable, the additional span vector u⃗'⃗_1...l. In the general case, the input to the classifier is:


    x⃗ = [u⃗_i ; u⃗_i+k ; u⃗'⃗_1 ; u⃗'⃗_l]


where each of u⃗_i+k, u⃗'⃗_1, and u⃗'⃗_l can be empty vectors in the cases of single word spans or no additional inputs.

The classifier output is defined as:


    o⃗ = softmax(W ·ReLU(Dropout(h(x⃗))))


where h is a 300-dimensional hidden layer, the dropout probability is 0.2, W ∈ℛ^k × 300, and k is the number of class labels for the specific task. 



  
Implementation Details. We implemented the models using the AllenNLP library <cit.> which is based on the PyTorch framework <cit.>. We train them for up to 500 epochs, stopping early if the validation performance doesn't improve in 20 epochs.

The phrase type model is a sequence tagging model that predicts a label for each embedded (potentially encoded) word w_i. During decoding, we enforce a single constraint that requires that a  tag must precede  tag(s). 







§ BASELINES






 §.§ Human Performance



The human performance on each task can be used as a performance upper bound which shows both the inherent ambiguity in the task, as well as the limitations of the particular dataset. We estimated the human performance on each task by sampling and re-annotating 100 examples from each test set. 

The annotation was carried out in Amazon Mechanical Turk. We asked 3 workers to annotate each example, taking the majority label as the final prediction. To control the quality of the annotations, we required that workers must have an acceptance rate of at least 98% on at least 500 prior HITs, and had them pass a qualification test. 

In each annotation task, we showed the workers the context sentence with the target span highlighted, and asked them questions regarding the target span as exemplified in Table <ref>. In addition to the possible answers given in the table, annotators were always given the choice of “I can't tell” or “the sentence does not make sense”. 

Of all the annotation tasks, the LVC classification task was more challenging and required careful examination of the different criteria for LVCs. In the example given in Table <ref> with the candidate LVC “make a decision”, we considered a worker's answer as positive (LVC) either if: 1) the worker indicated that “make a decision” does not describe an action of “making something” AND that the essence of “make a decision” is in the word “decision”; or 2) if the worker indicated that “make a decision” can be replaced in the given sentence with “decide” without changing the meaning of the sentence. The second criterion was given in the original guidelines of tu-roth:2011:MWE. The replacement verb “decide” was selected as it is linked to “decision” in WordNet in the derivationally-related relation. 

We didn't compute the estimated human performance on the phrase type task, which is more complicated and requires expert annotation.



 §.§ Majority Baselines



We implemented three majority baselines: 


    
  * Majority_ALL is computed by assigning the most common label in the training set to all the test items. Note that the label distribution may be different between the train and test sets, resulting in accuracy < 50% even for binary classification tasks.
    
  * Majority_1 assigns for each test item the most common label in the training set for items with the same first constituent. For example, in the VPC classification task, it classifies get through as positive in all its contexts because the verb get appears in more positive than negative examples.
    
  * Majority_2 symmetrically assigns the label according to the last (typically second) constituent. 






§ RESULTS






Table <ref> displays the best performance of each model family on the various tasks. 



  
Representations. The general trend across tasks is that the performance improves from the majority baselines through word embeddings and to the contextualized word representations, with a large gap in some of the tasks. Among the contextualized word embeddings, BERT performed best on 4 out of 6 tasks, with no consistent preference between ELMo and the OpenAI GPT. The best 
word embedding representations were GloVe (4/6) followed by fastText (2/6). 



  
Phenomena. The gap between the best model performance (achieved by the contextualized representations) and the estimated human performance varies considerably across tasks. The best performance in NC Literality is on par with human performance, and only a few points short of that in VPC Classification and LVC Classification. This is an evidence for the utility of contextualized word embeddings in detecting meaning shift, which has positive implications for the yet unsolved problem of detecting MWEs. 

Conversely, the gap between the best model and the human performance is as high as 23.5 and 21.3 points in NC Relations and AN Attributes, respectively, suggesting that tasks requiring revealing implicit meaning are more challenging to the existing representations.



  
Layer. Table <ref> elaborates on the best setting for each representation on the various tasks. In most cases, there was a preference to learning a scalar mix of the layers rather than using only the top layer. We extracted the learned layer weights for each of the  models, and found that the model usually learned a balanced mix of the top and bottom layers. 



  
Encoding. We did not find one encoding setting that performed best across tasks and contextualized word embedding models. Instead, it seems that tasks related to meaning shift typically prefer  or no encoding, while tasks related to implicit meaning performed better with either  or no encoding. 

When it comes to word embedding models, Table <ref> shows that  was preferred more often. This is not surprising. A contextualized word embedding of a certain word is, by definition, already aware of the surrounding words, obviating the need for a second layer of order-aware encoding. A word embedding based model, on the other hand, must rely on a biLSTM to learn the same. 



  
 Finally, the best settings on the Phrase Type task use  across representations. It may suggest that predicting a label for each word can benefit from a more structured modelling of word order. Looking into the errors made by the best model (BERT+All+biLM) reveals that most of the errors were predicting , i.e. missing the occurrence of a phrase. With respect to specific phrase types, near perfect performance was achieved among the more syntactic categories. Specifically, auxiliary (“Did they think we were [going to] feel lucky to get any reservation at all?”), adverbs (“any longer”), and determiners (“a bit”). 
In accordance to the VPC Classification task, the VPC label achieved 85% accuracy. 10% were missed (classified as ) and 5% were confused with a “weak” MWE. Two of the more difficult types were “weak” MWEs (which are judged as more compositional and less idiomatic) and idiomatic verbs. The former achieved accuracy of 22% (68% were classified as ) and the latter only 8% (62% were classified as ). Overall it seems that the model relied mostly on syntactic cues, failing to recognize semantic subtleties such as idiomatic meaning and level of compositionality. 





§ ANALYSIS


We focus on the contextualized word embeddings, and look into the representations they provide.







 §.§ Meaning Shift





  
Does the representation capture VPCs? The best performer on the VPC Classification task was the BERT+All+Att. To get a better idea of the signal that BERT contains for VPCs, we chose several ambiguous verb-preposition pairs in the dataset. We define a verb-preposition pair as ambiguous if it appeared in at least 8 examples as a VPC and at least 8 examples as a non-VPC. For a given pair we computed the BERT representations of the sentences in which it appears in the dataset, and, similarly to the model, we represented the pair as the concatenation of its word vectors. In each vector we averaged the layers using the weights learned by the model. Finally, we projected the computed vectors into 2D space using t-SNE <cit.>. Figure <ref> demonstrates 4 example pairs. The other pairs we plotted had similar t-SNE plots, confirming that the signal for separating different verb usages comes directly from BERT.  



  
Non-literality as a rare sense. nunberg1994idioms considered some non-literal compounds as “idiosyncratically decomposable”, i.e. which can be decomposed to possibly rare senses of their constituents, as in considering bee to have a sense of “competition” in spelling bee and crocodile to stand for “manipulative” in crocodile tears. Using this definition, we could possibly use the NC literality data for word sense induction, in which recent work has shown that contextualized word representations are successful <cit.>.  We are interested in testing not only whether the contextualized models are capable of detecting rare senses induced by non-literal usage, which we have confirmed in Section <ref>, but whether they can also model these senses. To that end, we sample target words that appear in both literal and non-literal examples, and use each contextualized word embedding model as a language model to predict the best substitutes of the target word in each context. Table <ref> exemplifies some of these predictions. 

Bold words are words judged reasonable in the given context, even if they don't have the exact same meaning as the target word. It is apparent that there are more reasonable substitutes for the literal examples, across models (left part of the table), but BERT performs better than the others. The OpenAI GPT shows a clear disadvantage of being uni-directional, often choosing a substitute that doesn't go well with the next word (“a train to from”). 

The success is only partial among non-literal examples. While some correct substitutes are predicted for (guilt) trip, the predictions are much worse for the other examples. The meaning of diamond in diamond wedding is “60th”, and ELMo makes the closest prediction, 10th (which would make it “tin wedding”). 400th is a borderline prediction, because it is also an ordinal number, but an unreasonable one in the context of years of marriage. 

Finally, the last example snake oil is unsurprisingly a difficult one, possibly “non-decomposable” <cit.>, as both constituents are non-literal. Some predicted substitutes, rogue and charmer, are valid replacements for the entire noun compound (e.g. “you are a rogue salesman”). Others go well with the literal meaning of snake creating phrases denoting concepts which can indeed be sold (snake egg, snake skin). 

Overall, while contextualized representations excel at detecting shifts from the common meanings of words, their ability to obtain meaningful representations for such rare senses is much more limited. 



 §.§ Implicit Meaning



The performance of the various models on the tasks that involve revealing implicit meaning are substantially worse than on the other tasks. In NC Relations, ELMo performs best with the biLM-encoded model using only the top layer of the representation, surpassing the majority baseline by only 4.3 points in accuracy. The best performer in AN Attributes is BERT, with no encoding and using all the layers, achieving accuracy of 65.1%, well above the majority baseline (50%). 



We are interested in finding out where the knowledge of the implicit meaning originates. Is it encoded in the phrase representation itself, or does it appear explicitly in the context sentences? Finally, could it be that the performance gap from the majority baseline is due to the models learning to recognize which paraphrases are more probable than others, regardless of the phrase itself?

To try answer this question, we performed ablation tests for each of the tasks, using the best performing setting for each (ELMo+Top+biLM for NC Relations and BERT+All+None for AN Attributes). We trained the following models (-X signifies the ablation of the X feature):


    
  * -Phrase: where we mask the phrase in its context sentence, e.g. replacing “Today, the house has become a wine bar or bistro called Barokk” with “Today, the house has become a something or bistro called Barokk”. Success in this setting may indicate that the implicit information is given explicitly in some of the context sentences.[A somewhat similar phenomenon was recently reported by senaldi2019neural. Their model managed to distinguish idioms from non-idioms, but their ablation study showed the model was in fact learning to distinguish between abstract contexts (in which idioms tend to appear) and concrete ones.]
    
    
  * -Context: the out-of-context version of the original task, in which we replace the context sentence by the phrase itself, as in setting it to “wine bar”. Success in this setting may indicate that the phrase representation contains this implicit information.

    
  * -(Context+Phrase): in which we omit the context sentence altogether, and provide only the paraphrase, as in “bar where people buy and drink wine”. Success in this setting may indicate that negative sampled paraphrases form sentences which are less probable in English.


Table <ref> shows the results of this experiment. A first observation is that the full model performs best on both tasks, suggesting that the model captures implicit meaning from various sources. In the NC Relations, all variants perform on par or worse than the majority baseline, achieving a few points less than the full model. In the AN Attributes task it is easier to see that the phrase (AN) is important for the classification, while the context is secondary. 



§ RELATED WORK




  
Probing Tasks. One way to test whether dense representations capture a certain linguistic property is to design a probing task for this property, and build a model that takes the tested representation as an input. This kind of “black box” testing has become popular recently. adi2016fine studied whether sentence embeddings capture properties such as sentence length and word order. conneau-EtAl:2018:Long extended their work with a large number of sentence embeddings, and tested various properties at the surface, syntactic, and semantic levels. Others focused on intermediate representations in neural machine translation systems <cit.>, or on specific linguistic properties such as agreement <cit.>, and tense <cit.>. 

More recently, tenney2018what and liu2019linguistic each designed a suite of tasks to test contextualized word embeddings on a broad range of sub-sentence tasks, including part-of-speech tagging, syntactic constituent labeling, dependency parsing, named entity recognition, semantic role labeling, coreference resolution, semantic proto-role, and relation classification. tenney2018what found that all the models produced strong representations for syntactic phenomena, but gained smaller performance improvements upon the baselines in the more semantic tasks. liu2019linguistic found that some tasks (e.g., identifying the tokens that comprise the conjuncts in a coordination construction) required fine-grained linguistic knowledge which was not available in the representations unless they were fine-tuned for the task. To the best of our knowledge, we are the first to provide an evaluation suite consisting of tasks related to lexical composition. 



  
Lexical Composition. There is a vast literature on multi-word expressions in general <cit.>, and research focusing on noun compounds <cit.>, adjective-noun compositions <cit.>, verb-particle constructions <cit.>, and light verb constructions <cit.>.

In recent years, word embeddings have been used to predict the compositionality of phrases <cit.>, and to identify the implicit relation in adjective-noun compositions <cit.> and in noun compounds <cit.>. 

pavlick-callisonburch:2016:P16-1 created a simpler variant of the recognizing textual entailment task <cit.> that tests whether an adjective-noun composition entails the noun alone and vice versa in a given context. They tested various standard models for RTE and found that the models performed poorly with respect to this phenomenon. To the best of our knowledge, contextualized word embeddings haven't been employed for tasks related to lexical composition yet.



  
Phrase Representations. With respect to obtaining meaningful phrase representations, there is a prominent line of work in learning a composition function over pairs of words. mitchell2010composition suggested simple composition via vector arithmetics. baroni-zamparelli:2010:EMNLP and later maillard-clark:2015:CoNLL treated adjectival modifiers as functions that operate on nouns and change their meanings, and represented them as matrices. zanzotto2010estimating and dinu-pham-baroni:2013:CVSC extended this approach and composed any two words by multiplying each word vector by a composition matrix. These models start by computing the phrases' distributional representation (i.e. treating it as a single token) and then learning a composition function that approximates it. 

The main drawbacks of this approach are that it assumes compositionality and that it operates on phrases with a pre-defined number of words. Moreover, we can expect the resulting compositional vectors to capture properties inherited from the constituent words, but it is unclear whether they also capture new properties introduced by the phrase. For example, the compositional representation of olive oil may capture properties like green (from olive) and fat (from oil), but would it also capture properties like expensive (a result of the extraction process)?

Alternatively, other approaches were suggested for learning general phrase embeddings, either using direct supervision for paraphrase similarity <cit.>, indirectly from an extrinsic task <cit.>, or in an unsupervised manner by extending the word2vec objective <cit.>. While they don't have constrains on the phrase length, these methods still suffer from the two other drawbacks above: they  assume that the meaning of the phrase can always be composed from its constituent meanings, and it is unclear whether they can incorporate implicit information and new properties of the phrase. We expected that contextualized word embeddings, which assign a different vector for a word in each given context, would address at least the first issue by producing completely different vectors to literal vs. non-literal word occurrences. 




§ DISCUSSION AND CONCLUSION


We showed that contextualized word representations perform generally better than static word embeddings on tasks related to lexical composition. However, while they are on par with human performance in recognizing meaning shift, they are still far from that in revealing implicit meaning. This gap may suggest a limit on the information that distributional models currently provide about the meanings of phrases. 

Going beyond the distributional models, an approach to build meaningful phrase representations can get some inspiration from the way that humans process phrases. A study on how L2 learners process idioms found that the most common and successful strategies were inferring from the context (57% success) and relying on the literal meanings of the constituent words (22% success) <cit.>. As opposed to distributional models that aim to learn from a large number of (possibly noisy and uninformative) contexts, the sentential contexts in this experiment were manually selected, and a follow up study found that extended contexts (stories) help the interpretation further <cit.>. The participants didn't simply rely on adjacent words or phrases, but also employed reasoning. For example, in the sentence “Robert knew that he was robbing the cradle by dating a sixteen-year-old girl”, the participants inferred that 16 is too young to date, combined it with the knowledge that cradle is where a baby sleeps, and concluded that rob the cradle means dating a very young person. This level of context modeling seems to be beyond the scope of current text representations. 

We expect that improving the ability of representations to reveal implicit meaning will require training them to handle this specific phenomenon. Our evaluation suite, the data and code will be made available. It is easily extensible, and may be used in the future to evaluate new representations for their ability to address lexical composition.




§ ACKNOWLEDGMENTS


This work was supported in part by an Intel ICRI-CI grant, the Israel Science Foundation grant 1951/17, the German Research Foundation through and the German-Israeli Project Cooperation (DIP, grant DA 1600/1-1). Vered is also supported by the Clore Scholars Programme (2017).


acl_natbib

