CASPR: Automated Evaluation Metric for Contrastive Summarization
Nirupan Ananthamurugan Dat Duong Philip George Ankita Gupta Sandeep Tata Beliz Gunel


    Air-taxi trajectory optimization with
 aerodynamic and motor models
    
        A version of this paper was originally presented at the AIAA SciTech 2023 FORUM held in National Harbor, MD on January 23-27, 2023.
        Paper number: AIAA 2023-0324. 
        <https://doi.org/10.2514/6.2023-0324>
    

    
    Nicholas C. Orndorff[PhD Student, Department of Mechanical and Aerospace Engineering] and
    John T. Hwang[Assistant Professor, Department of Mechanical and Aerospace Engineering]

    
====================================================================================================================================================================================================================================================================================================================








Summarizing comparative opinions about entities (e.g., hotels, phones) from a set of source reviews, often referred to as contrastive summarization, can considerably aid users in decision making. However, reliably measuring the contrastiveness of the output summaries without relying on human evaluations remains an open problem. Prior work has proposed token-overlap based metrics, Distinctiveness Score <cit.>, to measure contrast which does not take into account the sensitivity to meaning-preserving lexical variations. In this work, we propose an automated evaluation metric   to better measure contrast between a pair of summaries. Our metric is based on a simple and light-weight method that leverages natural language inference (NLI) task to measure contrast by segmenting reviews into single-claim sentences and carefully aggregating NLI scores between them to come up with a summary-level score. We compare  with Distinctiveness Score and a simple yet powerful baseline based on BERTScore. Our results on a prior dataset CoCoTRIP <cit.> demonstrate that  can more reliably capture the contrastiveness of the summary pairs compared to the baselines.














§ INTRODUCTION





Consider the familiar problem of using the web to compare two entities A and B while shopping for products ("Which phone better suits my needs?"), planning travel ("Which hotel should I book for my anniversary trip?"), or even looking at universities ("Which university is better for my career?"). Parsing subjective information from multiple reviews to identify contrastive opinions ("the room had modern decor" vs "the room's decor was outdated") is time-consuming. Therefore, the problem of generating summaries containing comparative opinions of two entities, or contrastive summaries, is of practical importance.

Given reviews of two entities A and B, the task of contrastive summarization <cit.> involves generating comparative summaries conditioned on both entities, and containing no overlapping information. For each pair of entities A and B, summary A ∖ B contains opinions in A that contrast with or are not present in B. This is illustrated in Figure <ref>.

However, there currently exists no reliable automated evaluation metric to measure this contrast, which is an obstacle to further work on this task without conducting large-scale human evaluations. Distinctiveness Score (DS) <cit.> treats token overlap between comparative summaries as a proxy for contrast. In our work, we demonstrate that this approach can have considerable limitations. We propose an automated evaluation metric, , to measure contrast between a pair of summaries obtained via the contrastive summarization task.  evaluates the logical relationships between pairs of summaries after decomposing summaries into single-claim sentences, employing an out-of-the-box NLI model to compute scores between each pair of sentences from the two summaries, and carefully aggregating scores to come up with a single contrastiveness score for a pair of summaries. We demonstrate that  can capture logical contrast beyond lexical and semantic variations on the CoCoTrip dataset, unlike existing baselines. We discuss possible future lines of investigation in this direction and release our modified dataset.




§ APPROACH

In this section, we describe the baselines, DS <cit.> and BS^-1, and how our proposed metric  measures logical contrast beyond lexical and semantic variations.


















 §.§ Baselines



  §.§.§ Distinctiveness Score

Distinctiveness Score (DS) <cit.> was introduced to measure contrast between three summaries - two contrastive summaries A∖ B and B∖ A, and a common summary A ∩ B. 

    DS = 1 - ∑_(y,z)∈Υ̂^2|𝒲_y∩𝒲_z| - 2 |⋂_y ∈Υ̂W_y |/|⋃_y ∈Υ̂W_y |,

Where Υ̂:={A∖ B, B ∖ A, A ∩ B}, W_y represents the token bag of summary y ∈Υ̂, and Υ̂^(2) represents 2-subsets of Υ̂.
A ∩ B contains opinions that are common between the entities A and B. For simplicity, we only consider the two contrastive summaries and hence modify the definition of the DS to remove the common summary from the set:

    DS = 1 - |𝒲_A ∖ B∩𝒲_B ∖ A|/|𝒲_A ∖ B∪𝒲_B ∖ A|,

The original DS simply measures the lexical overlap without differentiating between contrastive and common summaries; in fact, it can be extended to multiple entities and summaries. Therefore, removing the common summary from the equation does not alter the calculation or the definition of the DS.
The lower the token overlap, the higher the value of DS. It is scaled to be within [0,100] with higher numbers indicating higher contrast (low lexical overlap). As illustrated in Table <ref>, the reliance on token overlap leads to an erroneous DS score in cases where text is compared to its paraphrase.

























  §.§.§ Inverted BERTscore (BS^-1)

We include a simple but powerful baseline that we refer to as inverted BERTScore (BS^-1). BERTScore was introduced <cit.> to measure semantic similarity by aggregating cosine distances between contextual embeddings of generated and reference sequences. We use the HuggingFace implementation[<https://huggingface.co/spaces/evaluate-metric/bertscore>] which outputs a similarity score in the [0,1] range. We use the F1 score of BERTScore as the scores are symmetrical with order of the inputs. We invert and rescale this to get BS^-1 = 100* (1 - BERTScore) as a measure of contrastiveness, with higher values indicating higher contrast. We compute BS^-1 on summaries S^A ∖ B, S^B ∖ A as follows,





    BS^-1 = 1 - BS(S^A ∖ B, S^B ∖ A)





BS^-1 is in the range [0,100] with higher numbers indicating higher contrast. BS^-1 is more robust to lexical changes than DS due to its use of BERT word embeddings, as evidenced by our example in Table <ref>. However, it is not sensitive to sentence-level logical relationships; concretely, as in Table <ref>, it fails to detect that the addition of a single token "not" has created a logically contradictory sentence, and therefore assigns a low contrast score of 26.






















 §.§ 


We propose a metric  based on a method that evaluates the logical relationships between pairs of sentences, assigns a score to these logical relationships, and finally aggregates them across all sentence pairs in the two contrastive summaries.






Natural Language Inference (NLI) We use the textual entailment task, also referred to as NLI, to compare the logical relationship between two sentences. More formally, given a premise and a hypothesis, an NLI model determines the inference relationship (entailment, contradiction, neutral) between them. For each pair, we consider the entailment relationship in both directions. For our experiments we use off-the-shelf NLI model: [<https://huggingface.co/roberta-large-mnli>] <cit.> 



Forming Single-Claim Sentences Our dataset contains long sentences embodying multiple claims, which is commonplace in reviews. An example complex-claim sentence would be "The hotel's breakfast is included in the room's price, but a little expensive." which we break down to "The hotel's breakfast is included in the room's price." and "The hotel's breakfast is a little expensive." This process helps us determine the fine-grained logical relationships between sentences with NLI models more reliably. We decompose the original sentences using  text generation model from OpenAI with few-shot prompting. Refer to Appendix <ref> for the specific prompt we used, along with the output single-claim sentence, as well as details on other hyperparameters.


Scoring and Aggregation For two contrastive summaries S^A∖ B, S^B∖ A, CASPR is computed as the average contrastiveness score across all constituent sentences in these summaries. To score a single constituent sentence s_i^A ∖ B from summary S^A∖ B, we compare s_i^A ∖ B with all sentences from the other summary s^B ∖ A, and compute a contrastiveness score depending on its comparison-level labels with these sentences.

Consider a single comparison of our sentence s_i^A ∖ B with another sentence s_j^B ∖ A from summary S^B∖ A. To get a label for this comparison, we evaluate the NLI label in both directions. The intuition for this is that NLI is directional, since for the same pair of sentences, swapping the hypothesis and premise changes the relationship between them. Considering this, we assign the label for a single comparison as follows:

    
  * If both directions are `neutral', then the label for that comparison is `neutral'.
    
  * If the labels are (`neutral', `contradiction') or (`contradiction', `neutral'), the label for this comparison is `contradiction'.
    
  * If the labels are (`neutral', `entailment') or (`entailment', `neutral'), the label for this comparison is `entailment'.
    
  * If the labels are (`contradiction', `entailment') or (`entailment', `contradiction'), the label for this comparison is `neutral'. This was a design choice based on our observation that there were very few such cases. In those few such cases, human examination revealed the model had incorrectly classified neutral statements as contradiction/entailment. 



We now have a single comparison label for s_i^A ∖ B with s_j^B ∖ A. We repeat the above process across all comparisons between s_i^A ∖ B and sentences from summary S^B∖ A. We process this set of labels according to the following conditions, to get a single score in the range [0,1] for our sentence s_i^A ∖ B.

    
  * Case 1: All Comparisons are Neutral - If s_i^A ∖ B was neutral to all sentences in the other summary, we take this to mean that s_i^A ∖ B mentions an aspect that is not mentioned in the other summary. We consider this a valid case of contrast, and assign it a score of +1.

If Case 1 is not true, then this means that there are some contradictions and/or entailments, i.e. there are some comparisons which are discussing the same aspect, and have values that agree/disagree with each other. In this case, we ignore the neutralities and count the number of contradictions and entailments.

    
  * Case 2: Number of Contradictions > Number of Entailments - We take this to mean that the sentence is overall contrastive w.r.t the other summary, and score it +1.
    
  * Case 3: Number of Entailments >= Number of Contradictions - We take this to mean that the sentence is overall similar w.r.t the other summary, and score it -1.

Note from the above, that in cases where the tally of entailments and contradictions is tied, we bias the score in favour of entailments; we found that the model was less sensitive to entailments in practice, and we decided to conservatively score ties in favor of similarity rather than contrast to offset this. We now have a +1/-1 score for our sentence s_i^A ∖ B. We now repeat this process for all sentences across both summaries S^A ∖ B and S^B ∖ A and take the mean score to get CASPR.






























§ EXPERIMENTS



 §.§ Dataset: CoCoTrip

We use the CoCoTrip dataset introduced by <cit.> which is designed to evaluate contrastive summarization methods. CoCoTrip consists of two contrastive summary sets S^A ∖ B, S^B ∖ A, for a pair of entities A and B summarized from review sets R^A and R^B. The entities are hotels drawn from the TripAdvisor corpus <cit.>, with 8 reviews sampled per entity, for a total of 48 entity pairs. For each pair, they source reference summaries from three different annotators.
Note that CoCoTrip also provides one common summary for each entity pair which we do not consider as we are interested in evaluating contrastiveness.



 §.§ Experimental Setup

From the CoCoTrip dataset, we have reference summaries S^A ∖ B, S^B ∖ A from three annotators for each entity pair. The summaries S^A ∖ B_1, S^B ∖ A_1 from the first annotator form our base dataset - the Reference Contrastive dataset. Similarly, we pair reference summaries from the first and second annotators to obtain a dataset of summaries S^A ∖ B_1, S^A ∖ B_2 - the Reference Similar dataset, as they are essentially summaries of the same entity by different annotators. We construct synthetic data with paraphrasing and logical negation that we describe below. We provide more details for each experiment along with examples from each constructed dataset in Appendix <ref>.


Logical Negations for Synthetic High Contrast To measure the ability of evaluation metrics to capture high contrast, we construct a synthetic dataset with logical negations. In particular, we manually negate all sentences in summary S^A ∖ B_1 to obtain a dataset consisting of summaries S^A ∖ B_1,  S^A ∖ B_1 for each entity pair. We refer to this dataset as Synthetic High Contrast dataset. 


Self-Paraphrasing for Synthetic Low Contrast We construct a Synthetic Low Contrast dataset by paraphrasing reference summaries S^A ∖ B. The resulting dataset consisting of summaries S^A ∖ B, P(S^A ∖ B) should exhibit low contrast. We paraphrase each sentence in the summary using OpenAI's . Implementation details and examples are provided in Appendix <ref>.


For an ideal metric, we expect the following: 

    
  * Close to 0 score on Synthetic Low Contrast dataset (S^A ∖ B, P(S^A ∖ B)) as summaries are semantically equivalent.
    
  * Close to 100 score on Synthetic High Contrast dataset (S^A ∖ B,  S^A ∖ B) as summaries are logical contradictions of each other.
    
  * Ranking from low to high scores to be Synthetic Low Contrast (S^A ∖ B, P(S^A ∖ B)), Reference Similar (S^A ∖ B_1, S^A ∖ B_2), Reference Contrastive (S^A ∖ B, S^B ∖ A), and Synthetic High Contrast (S^A ∖ B,  S^A ∖ B) datasets with meaningful separation in scores.



We publish our Synthetic Low Contrast and Synthetic High Contrast datasets in the [<https://github.com/niru-umass-dev/caspr>].














 §.§ Results






We use a non-parametric bootstrap <cit.>
to infer confidence intervals for each method. We utilize 10^4 bootstrap samples of summary pairs, to report 95% bootstrap confidence intervals (CI), via the normal interval method <cit.>. In Table <ref>, we see that all metrics register high scores on the Reference Contrastive dataset (S^A ∖ B, S^B ∖ A). On the Synthetic High Contrast dataset (S^A ∖ B,   S^A ∖ B),  increases to 98.1 and is a near-ideal score. However, both DS(44.5) and BS^-1(50.6) drop, which supports our assertion that they are not sensitive to logical contrast. In Figure <ref>, we see that  moves from low to high scores as we traverse from Synthetic Low Contrast to Synthetic High Contrast, for which it has the closest scores to 0 and 100 respectively. It has a meaningful separation of scores between all four datasets, since it can not only capture logical variations (where DS and BS^-1 clearly both fail) but can also capture semantic and lexical variations introduced by the other datasets.  DS and BS^-1 have a much lower separation and counter-intuitively drop for Synthetic High Contrast, compared to Reference Contrastive.
    


§ RELATED WORK

Recently, there has been considerable interest in developing automated evaluation metrics in order to ensure faster and higher quality iteration on various ML tasks across different domains <cit.>. Few recent works <cit.> develop contrastive summarization methods, but do not offer automated evaluation metrics. Most relevant to our work, collaborative decoding method <cit.> was proposed which contrasts token probability distributions to generate two contrastive summaries. They conduct automated evaluation of their summarization quality using ROUGE 1/2/L F1 scores <cit.>, BERTScore <cit.>, and Distinctiveness Score. However, as illustrated in Table <ref>, these metrics which are based on lexical overlap and BERTScore, do not capture simple logical relationship among sentence pairs (e.g., negations). Finally, NLI-based methods has been shown effective for evaluating factual consistency of single entity summarization methods <cit.>. To the best of our knowledge, ours is the first work to utilize NLI-based methods for quantifying contrast among a pair of contrastive summaries.











§ CONCLUSION

In this paper, we propose an automated evaluation metric  for the contrastive summarization task.  evaluates the logical relationships between pairs of summaries after decomposing summaries into single-claim sentences, employing a textual entailment model, and carefully aggregating scores to come up with a summary-level contrastiveness score. We demonstrate that  can capture logical contrast beyond lexical and semantic variations on CoCoTrip dataset, unlike existing baselines. Regarding ease of implementation, our metric uses out-of-the-box models, hence does not need any fine-tuning.



§ LIMITATIONS

We list some of the limitations of our study which researchers and practitioners would hopefully benefit from when interpreting our results. We used  for NLI which is a small model, and  for forming single-claim sentences to demonstrate the benefits of using an approach based on NLI to measure contrast. However, we acknowledge that using state-of-the-art large language models such as  for both of these steps might yield much better results.
Also, we acknowledge that CASPR specifically focuses on measuring contrast among a pair of summaries. We consider our work to be complementary to other recent efforts in this research area, such as measuring factuality of summaries with respect to the source reviews  
<cit.> or measuring how well a metric captures the popular opinions or overall sentiment on various aspects mentioned in a summary <cit.>.

plainnat 






§ SENTENCE DECOMPOSITION


We used the  model from the OpenAI Chat Completion API. Figure <ref> shows the specific few-shot prompt we used, along with an example of an input complex-claim sentence and output simple-claim sentences. Table <ref> shows an example summary S_1^A ∖ B from the CoCoTrip dataset and the corresponding decomposed summary formed of single-claim sentences.
We set  and  to encourage the model to stick close to the structure of the original sentence. Note that although  is a chat-completion model[https://platform.openai.com/docs/models/gpt-3-5], we treated every complex sentence as a new chat, and included only the current sentence in each API call without appending prior input and responses.



§ EXPERIMENTAL DATASETS









Table <ref> shows the different experimental datasets referred to in our paper and Table <ref> provides a selection of examples from each of these datasets.



 §.§ Paraphrasing

For paraphrasing, we use the text-davinci-003
large language model from OpenAI. The prompt
used for paraphrasing was just 'Paraphrase this' and the input summary. We set  and .



 §.§ Negations

For creating negations, we first split each sentence using the process discussed with , since it is non trivial to negate complex-claim sentences. For each single-claim sentence for a summary we obtain negated sentences by way of manual annotation. This achieved by adding modifiers ('not'), substituting complements of quantifiers ('everyone' to 'no one'), or using antonyms ('close' to 'far away'). Then we re-combine negated simple-claim sentences into complex-claim sentences using a rule-based method of simple conjunctions, since our baselines operate on complex-claim sentences.




§ CALCULATING 



We describe here how we calculate the  score after obtaining single-claim sentences by the process described in Appendix <ref>. We compare each single-claim sentence s^A ∖ B_i in summary S^A ∖ B with every single-claim sentence s^B ∖ A_j in summary S^B ∖ A. For each comparison (s^A ∖ B_i,s^B ∖ A_j), we define the indicator function 1_cont for the 'CONTRADICTION' label as,

    1_cont = 
        
            1,    [t]
            NLI(s^A ∖ B_i → s^B ∖ A_j)= cont
    and NLI(s^B ∖ A_j → s^A ∖ B_i) ≠ent
    
    
    
        1,    [t]
            NLI(s^B ∖ A_j→ s^A ∖ B_i)= cont
    and NLI(s^A ∖ B_i → s^B ∖ A_j) ≠ent
    
    
    
            0,     otherwise

That is, if the NLI label for (s^A ∖ B_i,s^B ∖ A_j) in either direction is contradiction, and the label in the other direction is not entailment, the indicator function for contradiction has a value of 1. We get the indicator function 1_ent of the 'ENTAILMENT' label similarly. For the 'NEUTRAL' indicator function we have,

    1_neut = 
        
            1,    [t]
            NLI(s^A ∖ B_i → s^B ∖ A_j)= neut
    and NLI(s^B ∖ A_j→ s^A ∖ B_i)= neut
    
    
    
         1,    [t]
            NLI(s^A ∖ B_i → s^B ∖ A_j)= cont
    and NLI(s^B ∖ A_j→ s^A ∖ B_i)= ent
    
    
    
        1,    [t]
            NLI(s^A ∖ B_i → s^B ∖ A_j)= ent
    and NLI(s^B ∖ A_j→ s^A ∖ B_i)= cont
    
    
    
            0,     otherwise

We now aggregate the values of the indicator functions 1_cont,1_ent,1_neut over all sentences s^B ∖ A_j ∈ S^B ∖ A for the sentence s^A ∖ B_i to get the counts n^i_cont,n^i_neut,n^i_neut For example, we get n^i_cont as follows,

    n^i_cont = ∑_s^B ∖ A_j ∈ S^B ∖ A 1_cont (s^A ∖ B_i, s^B ∖ A_j)

We then get the label score l(s^A ∖ B_i|S^B ∖ A) of the sentence by comparing the aggregates in the following way,

    l(s^A ∖ B_i|S^B ∖ A)  =
        
            +1,     n^i_neut = |S^B ∖ A| 
    
    
            -1,     n^i_ent >= n^i_cont
    
    
            +1,     otherwise

where n^i_neut = |S^B ∖ A| denotes the case where all comparisons evaluate to 'neut' in both directions.
We them sum the label scores across all sentences of both summaries.

    L_AB = ∑_s^A ∖ B_i ∈ S^A ∖ B l(s^A ∖ B_i|S^B ∖ A)
    
        + ∑_s^B ∖ A_i ∈ S^B ∖ A
        l(s^B ∖ A_i|S^A ∖ B)

We then normalize L_AB across the total number of sentences across both summaries.

    L̂ = L_AB/|S^A ∖ B| + |S^B ∖ A|

L̂ lies in the range [-1,1]. We finally rescale it to lie in the range [0,100].

    = L̂ + 1/2 * 100
































§ SUPPLEMENTARY RESULTS: PARAPHRASING


    is reasonably robust to lexical changes that preserve meaning. Specifically, if we generate the paraphrases of Reference Contrastive summaries S^A ∖ B,S^B ∖ A to obtain Synthetic Contrastive summaries P(S^A ∖ B), P(S^B ∖ A), mean  scores shows negligible differences between the two datasets. BS^-1 and DS show similar differences. BS^-1(86%) outperforms (66%) and DS(51%) in regards to rank correlation between scores from these datasets. This is due to the original purpose of BERTScore as a similarly score between two sentences using semantic embeddings. An explanation of this dataset is included in Figure <ref>.















§ ERROR ANALYSIS OF NLI FOR 




An interesting scenario (Table <ref>)  where NLI fails is when the hypothesis is a common sense maxims. For e.g.  the hypothesis 'This hotel provides a stay' always evaluates to an entailment, no matter the premise (unless the premise explicitly contradicts it with 'this hotel does not provide a stay' which goes against commonsense). This leads to more entailments and low contrast scores on the Reference Contrastive dataset. This might be remedied with a stronger NLI model and a better sentence decomposition model.



