CAN: Constrained Attention Networks for Multi-Aspect Sentiment Analysis
Mengting Hu Shiwan Zhao Li Zhang Keke Cai Zhong Su Renhong Cheng Xiaowei Shen

On mutual information estimation for mixed-pair random variables
    Roberto Aloisio1,2roberto.aloisio@gssi.it
    
================================================================


Aspect level sentiment classification is a fine-grained sentiment analysis task. To detect the sentiment towards a particular aspect in a sentence, previous studies have developed various attention-based methods for generating aspect-specific sentence representations. However, the attention may inherently introduce noise and downgrade the performance. In this paper, we propose constrained attention networks (CAN), a simple yet effective solution, to regularize the attention for multi-aspect sentiment analysis, which alleviates the drawback of the attention mechanism. Specifically, we introduce orthogonal regularization on multiple aspects and sparse regularization on each single aspect. 
Experimental results on two public datasets demonstrate the effectiveness of our approach. We further extend our approach to multi-task settings and outperform the state-of-the-art methods.




§ INTRODUCTION

Sentiment analysis <cit.>, an important task in natural language understanding, receives much attention in recent years. Aspect level sentiment classification (ALSC) is a fine-grained sentiment analysis task, which aims at detecting the sentiment towards a particular aspect in a sentence. ALSC is especially critical for multi-aspect sentences which contain multiple aspects. A multi-aspect sentence can be categorized as overlapping or non-overlapping. A sentence is annotated as non-overlapping if any two of its aspects have no overlap. Our study found that around 85% of the multi-aspect sentences are non-overlapping in the two public datasets. Figure <ref> shows a simple example. The non-overlapping sentence contains two aspects. The aspect food is on the left side of the aspect service. Their distributions on words are orthogonal to each other. Another observation is that only a few words relate to the opinion expression in each aspect. As shown in Figure <ref>, only the word “good” is relevant to the aspect food and “ok" to service. The distribution of the opinion expression of each aspect is sparse. 

To detect the sentiment towards a particular aspect, previous studies <cit.> have developed various attention-based methods for generating aspect-specific sentence representations. In these works, the attention may inherently introduce noise and downgrade the performance <cit.> since the attention scatters across the whole sentence and is prone to attend on noisy words, or the opinion words from other aspects. Take Figure <ref> as an example, for the aspect food, we visualize the attention weights from the model <cit.>. Much of the attention focuses on the noisy word “sometimes", and the opinion word “ok" which is relevant to the aspect service rather than food.




To alleviate the above issue, we propose a model for multi-aspect sentiment analysis, which regularizes the attention by handling multiple aspects of a sentence simultaneously. Specifically, we introduce orthogonal regularization for attention weights among multiple non-overlapping aspects. The orthogonal regularization tends to make the attention weights of multiple aspects concentrate on different parts of the sentence with less overlap. We also introduce the sparse regularization, which tends to make the attention weights of each aspect concentrate only on a few words. We call our networks with such regularizations constrained attention networks (CAN). 


There have been some works on introducing sparsity in attention weights in machine translation <cit.> and orthogonal constraints in domain adaptation <cit.>. In this paper, we add both sparse and orthogonal regularizations in a unified form inspired by the work <cit.>.
The details will be introduced in Section <ref>.

In addition to aspect level sentiment classification (ALSC), aspect category detection (ACD) is another task of aspect based sentiment analysis.  ACD <cit.> aims to identify the aspect categories discussed in a given sentence from a predefined set of aspect categories (e.g., price, food, service). Take Figure <ref> as an example, aspect categories food and service are mentioned. We introduce ACD as an auxiliary task to assist the ALSC task, benefiting from the shared context of the two tasks. We also apply our attention constraints to the ACD task. By applying attention weight constraints to both ALSC and ACD tasks in an end-to-end network, we can further evaluate the effectiveness of CAN in multi-task settings.

In summary, the main contributions of our work are as follows:


  * We propose CAN for multi-aspect sentiment analysis. Specifically, we introduce orthogonal and sparse regularizations to constrain the attention weight allocation, helping learn better aspect-specific sentence representations. 


  * We extend CAN to multi-task settings by introducing ACD as an auxiliary task, and applying CAN on  both ALSC and ACD tasks. 


  * Extensive experiments are conducted on public datasets. Results demonstrate the effectiveness of our approach for aspect level sentiment classification.  




§ RELATED WORK


Aspect level sentiment analysis is a fine-grained sentiment analysis task. Earlier methods are usually based on explicit features <cit.>. 
With the development of deep learning technologies, various neural attention mechanisms have been proposed to solve this fine-grained task <cit.>. To name a few,  Wang2016Attention propose an attention-based LSTM network for aspect level sentiment
classification. 

Ma2017Interactive use the interactive attention
networks to generate the representations for targets and contexts separately. 

cheng2017aspect,ruder2016hierarchical both propose hierarchical neural network models for aspect level sentiment classification.  

wang2018learning employ a segmentation
attention based LSTM model for aspect level sentiment classification. All these works can be categorized as single-aspect sentiment analysis, which deals with aspects in a sentence separately, without considering the relationship between aspects. 


More recently, a few works have been proposed to take the relationship among multiple aspects into consideration. multi-aspect1-naacl make simultaneous classification of all aspects in a sentence using recurrent networks.  multi-aspect3-emnlp employ memory network to model the dependency of the target aspect with the other aspects in the sentence. multi-aspect2-emnlp design an aspect alignment loss to enhance the difference of the
attention weights towards the aspects which have
the same context and different sentiment polarities. In this paper, we introduce orthogonal regularization to constrain the attention weights of multiple non-overlapping aspects, as well as sparse regularization on each single aspect. 






Multi-task learning Caruana1997Multitask solves multiple learning tasks at the same time, achieving improved performance by exploiting commonalities and differences across tasks. Multi-task learning has been used successfully in many  machine learning applications. Huang2018Multitask learn both main task and auxiliary task jointly with shared representations, achieving improved performance in question answering. Toshniwal2017Multitask use low-level auxiliary tasks
for encoder-decoder based speech recognition, which suggests that the addition of auxiliary tasks can help in either optimization or generalization. yu2016learning use two auxiliary tasks to help induce a sentence embedding that works well across domains for sentiment classification. In this paper, we adopt the multi-task learning approach by using ACD as the auxiliary task to help the ALSC task. 






§ MODEL


We first formulate the problem. There are totally N predefined aspect categories in the dataset, A={A_1,...,A_N}. Given a sentence S={w_1, w_2, ..., w_L}, which contains K aspects A^s={A_1^s,...,A_K^s}, A^s⊆  A, the multi-task learning is to simultaneously solve the ALSC and ACD tasks, namely, the ALSC task predicts the sentiment polarity of each aspect A_k^s ∈ A^s, and the auxiliary ACD task checks each aspect A_n ∈ A to see whether the sentence S mentions it. Note that we only focus on aspect-category instead of aspect-term <cit.> in this paper.

We propose CAN for multi-aspect sentiment analysis, supporting both ALSC and ACD tasks by a multi-task learning framework. The network architecture is shown in Figure <ref>. We will introduce all components sequentially from left to right.



 §.§ Embedding and LSTM Layers

Traditional single-aspect sentiment analysis handles each aspect separately, one at a time. In such settings, a sentence S with K aspects will be copied to form K instances. For example, a sentence S contains two aspects: A_1^s with polarity p_1 and A_2^s with polarity p_2. Two instances, ⟨S, A_1^s, p_1⟩ and ⟨S, A_2^s, p_2⟩, will be constructed. Our multi-aspect sentiment analysis method handles multiple aspects together and takes the single instance ⟨S, [A_1^s, A_2^s], [p_1, p_2]⟩ as input. 

The input sentence {w_1, w_2, ..., w_L} is first converted to a sequence of vectors {v_1,v_2,...,v_L}, and the K aspects of the sentence are transformed to vectors {u_1^s,...,u_K^s}, which is a subset of {u_1,...,u_N}, the vectors of all aspect categories. The word embeddings of the sentence are then fed into an LSTM network <cit.>, which outputs hidden states H={h_1,h_2,...,h_L}. The sizes of the embedding and the hidden state are both set to be d.



 §.§ Task-Specific Attention Layer

The ALSC and ACD tasks share the hidden states from the LSTM layer, while compute their own attention weights separately. The attention weights are then used to compute aspect-specific sentence representations. 

ALSC Attention Layer
The key idea of aspect level sentiment classification is to learn different attention weights for different aspects, so that different aspects can concentrate on different parts of the sentence. 

We follow the approach in the work <cit.> to compute the attention. Particularly, given the sentence S with K aspects, A^s={A_1^s,...,A_K^s}, for each aspect A_k^s, its attention weights are calculated by:

    α_k = softmax(z^a^Ttanh(W_1^aH + W_2^a(u_k^s⊗e_L)))

where u_k^s is the embedding of the aspect A_k^s, e_L∈ℝ^L is a
vector of 1s, u_k^s⊗e_L is the operation repeatedly concatenating u_k^s for L times. W_1^a∈ℝ^d×d, W_2^a∈ℝ^d×d and z^a∈ℝ^d are the weight matrices.

ACD Attention Layer We treat the ACD task as multi-label classification problem for the set of N aspect categories. For each aspect A_n∈ A, its attention weights are calculated by:

    β_n = softmax(z^b^Ttanh(W_1^bH + W_2^b(u_n⊗e_L)))

where u_n is the embedding of the aspect A_n. W_1^b∈ℝ^d×d, W_2^b∈ℝ^d×d and z^b∈ℝ^d are the weight matrices.

The ALSC and ACD tasks use the same attention mechanism, but they do not share parameters. The reason to use separated parameters is that, for the same aspect, the attention of ALSC concentrates more on opinion words, while ACD focuses more on aspect target terms (see the attention visualizations in Section <ref>). 




 §.§ Regularization Layer

We simultaneously handles multiple aspects by adding constraints to their attention weights. Note that this layer is only available in the training stage, in which the ground-truth aspects are known for calculating the regularization loss, and then influence parameter updating in back propagation. While in the testing/inference stage, the true aspects are unknown and the regularization loss is not calculated so that this layer is omitted from the architecture.  

In this paper, we introduce two types of regularizations: the sparse regularization on each single aspect; the orthogonal regularization on multiple non-overlapping aspects. 



Sparse Regularization For each aspect, the sparse regularization constrains the distribution of the attention weights (α_k or β_n) to concentrate on less words. For simplicity, we use α_k as an example, α_k={α_k1,  α_k2, ..., α_kL}. To make α_k sparse, the sparse regularization term is defined as: 

    R_s = |∑_l=1^Lα_kl^2 - 1|


where ∑_l=1^Lα_kl=1 and α_kl>0. Since α_k is normalized as a probability distribution, L_1 norm is always equal to 1 (the sum of the probabilities) and does not work as sparse regularization as usual. Minimizing Equation <ref> will force the sparsity of α_k. It has the similar effect as minimizing the entropy of α_k, which leads to placing more probabilities on less words.

Orthogonal Regularization This regularization term forces orthogonality among attention weight vectors of multiple aspects, so that different aspects attend on different parts of the sentence with less overlap. Note that we only apply this regularization to non-overlapping multi-aspect sentences. Assume that the sentence S contains K non-overlapping aspects {A_1^s,...,A_K^s} and their attention weight vectors are {α_1,...,α_K}. We pack them together as a two-dimensional attention matrix M∈ℝ^K×L to calculate the orthogonal regularization term.

    R_o = ∥M^TM - I∥_2

where I is an identity matrix. In the resulted matrix of M^TM, each non-diagonal element is the dot product between two attention weight vectors, minimizing the non-diagonal elements will force orthogonality between corresponding attention weight vectors. The diagonal elements of M^TM are subtracted by 1, which are the same as R_s defined in Equation <ref>. As a whole, R_o includes both sparse and orthogonal regularization terms. 

Note that in the ACD task, we do not pack all the N attention vectors {β_1, ..., β_N} as a matrix. The sentence S contains K aspects. For simplicity, let {β_1, ..., β_K} be the attention vectors of the K aspects mentioned, while {β_K+1, ..., β_N} be the attention vectors of the N-K aspects not mentioned. We compute the average of the N-K attention vectors, denoted by β_avg. We then construct the attention matrix G={β_1, ..., β_K,β_avg}, G∈ℝ^(K+1)×L. The reason why we calculate β_avg is that if an aspect is not mentioned in the sentence, its attention weights often attend to meaningless stop words, such as “to”, “the”, “was”, etc. We do not need to distinguish among the N-K aspects not mentioned, therefore they can share stop words in the sentence by being averaged as a whole, which keeps the K aspects mentioned away from such stop words. 



 §.§ Task-Specific Prediction Layer

Given the attention weights of each aspect, we can generate aspect-specific sentence representation, and then make prediction for the ALSC and ACD tasks respectively.


    r_k^s=∑_l=1^Lα_klh_l
    r_n=∑_l=1^Lβ_nlh_l





ALSC Prediction The weighted hidden state is combined with the last hidden state to generate the final aspect-specific sentence representation.

    r_k^s=tanh(W_1^rh̅_k+W_2^rh_L)

where W_1^r∈ℝ^d×d and W_2^r∈ℝ^d×d. h̅_k =∑_l=1^Lα_klh_l is the weighted hidden state for aspect k. r_k^s is then used to make sentiment polarity prediction.

    ŷ_̂k̂=softmax(W_p^ar_k^s+b_p^a)

where W_p^a∈ℝ^d×c and b_p^a∈ℝ^c are the parameters of the projection layer, and c is the number of classes. 

For the sentence S with K aspects mentioned, we make K predictions simultaneously. That is why we call our approach multi-aspect sentiment analysis.  

ACD Prediction We directly use the weighted hidden state as the sentence representation for ACD prediction.

    r_n =h̅_n =∑_l=1^Lβ_nlh_l
 
We do not combine with the last hidden state h_L since the aspect may not be mentioned by the sentence. We make N predictions for all predefined aspect categories. 

    ŷ_̂n̂=sigmoid(W_p^br_n+b_p^b)

where W_p^b∈ℝ^d×1 and b_p^b is a scalar. 



 §.§ Loss

For the task ALSC, the loss function for the K aspects of the sentence S is defined by:

    L_a = -∑_k=1^K∑_cy_kclogŷ_̂k̂ĉ

where c is the number of classes. For the task ACD, as each prediction is binary classification problem, the loss function for the N aspects of the sentence S is defined by:

    L_b = -∑_n=1^N[y_nlogŷ_̂n̂ + (1-y_n)log(1-ŷ_̂n̂)]


We jointly train our model for the two tasks. The
parameters in our model are then trained by minimizing the combined loss function:

    L = L_a + 1/NL_b + λR

where R is the regularization term mentioned previously, which can be R_s or R_o. λ is the hyperparameter used for tuning the impact from regularization
loss to the overall loss. To avoid L_b overwhelming the overall loss, we divide it by the number of aspect categories.






§ EXPERIMENTS



 §.§ Datasets

We conduct experiments on two public datasets from SemEval 2014 task 4 <cit.> and SemEval 2015 task 12 <cit.> (denoted by Rest14 and Rest15 respectively). These two datasets consist of restaurant customer reviews with annotations identifying the mentioned aspects and the sentiment polarity of each aspect. To apply orthogonal regularization, we manually annotate the multi-aspect sentences with overlapping or non-overlapping[We will release the annotated dataset later.]. We randomly split the original training set into training, validation sets in the ratio 5:1, where the validation set is used to select the best model. We count the sentences of single-aspect and multi-aspect separately. Detailed statistics are summarized in Table <ref>. Particularly, 85.23% and 83.73% of the multi-aspect sentences are non-overlapping in Rest14 and Rest15, respectively. 









 §.§ Comparison Methods

Since we focus on aspect-category sentiment analysis, many works <cit.> which focus on aspect-term sentiment analysis are excluded.


  * LSTM: We implement the vanilla LSTM to model the sentence and use the average of all hidden states as the sentence representation. In this model, aspect information is not used.


  * AT-LSTM <cit.>: It adopts the attention mechanism in LSTM to generate a weighted representation of a sentence. The aspect embedding is used to compute the attention weights as in Equation <ref>. We do not concatenate the aspect embedding to the hidden state as in the work <cit.> and gain small performance improvement. We use this modified version in all experiments. 


  * ATAE-LSTM <cit.>: This method is an extension of AT-LSTM. In this model, the aspect embedding is concatenated to each word embedding of the sentence as the input to the LSTM layer. 




  * GCAE <cit.>: This state-of-the-art method is based on the convolutional
neural network with gating mechanisms, which is for both aspect-category and aspect-term sentiment analysis. We compare with its aspect-category sentiment analysis task.






 §.§ Our Methods

To verify the performance gain of introducing constraints on attention weights, we first create several variants of our model for single-task settings.





  * AT-CAN-R_s: Add sparse regularization R_s to AT-LSTM to constrain the attention weights of each single aspect. 


  * AT-CAN-R_o: Add orthogonal regularization R_o to AT-CAN-R_s to constrain the attention weights of multiple non-overlapping aspects. 



  * ATAE-CAN-R_s: Add R_s to ATAE-LSTM.


  * ATAE-CAN-R_o: Add R_o to ATAE-CAN-R_s.



We then extend attention constraints to multi-task settings, creating variants by different options: 1) no constraints, 2) adding regularizations only to the ALSC task, 3) adding regularizations to both tasks.



  * M-AT-LSTM: This is the basic multi-task model without regularizations.


  * M-CAN-R_s: Add R_s to the ALSC task in M-AT-LSTM.


  * M-CAN-R_o: Add R_o to the ALSC task in M-CAN-R_s.


  * M-CAN-2R_s: Add R_s to both tasks in M-AT-LSTM. 


  * M-CAN-2R_o: Add R_o to both tasks in M-CAN-2R_s.








 §.§ Implementation Details


We set λ=0.1 with the help of the validation set. All models are optimized by the Adagrad optimizer <cit.> with learning rate 0.01. Batch size is 25. We apply a dropout of p=0.7 after the embedding and LSTM layers. All words in the sentences are initialized with 300 dimension Glove Embeddings <cit.>. The aspect embedding matrix and parameters are initialized by sampling from a uniform distribution U(-ε,ε), ε=0.01. d is set as 300. The models are trained for 100 epochs, during which the model with the best performance on the validation set is saved. We also apply early stopping in training, which means that the training will stop if the performance on validation set does not improve in 10 epochs. 






 §.§ Results

Table <ref> and <ref> show our experimental results on the two public datasets for single-task and multi-task settings respectively. In both tables, “3-way” stands for 3-class classification (positive, neutral, and negative), and “Binary” for binary classification (positive and negative). The best scores are marked in bold. 


Single-task Settings Table <ref> shows our experimental results of ALSC in single-task settings. Firstly, we observe that by introducing attention regularizations (either R_s or R_o), most of our proposed methods outperform their counterparts. Particularly, AT-CAN-R_s and AT-CAN-R_o outperform AT-LSTM in all results; ATAE-CAN-R_s and ATAE-CAN-R_o also outperform ATAE-LSTM in 15 of 16 results. For example, in the Rest15 dataset, ATAE-CAN-R_o outperforms ATAE-LSTM by up to 5.39% of accuracy and 6.46% of the F1 score in the 3-way classification. Secondly, regularization R_o achieves better performance improvement than R_s in all results. This is because R_o includes both orthogonal and sparse regularizations for non-overlapping multi-aspect sentences. Thirdly, our approaches, especially ATAE-CAN-R_o, outperform the state-of-the-art baseline model GCAE. Finally, the LSTM method outputs the worst results in all cases, because it can not distinguish different aspects.



Multi-task Settings Table <ref> shows experimental results of ALSC in multi-task settings. We first observe that the overall results in multi-task settings outperform the ones in single-task settings, which demonstrates the effectiveness of multi-task learning by introducing the auxiliary ACD task to help the ALSC task. Second, in almost all cases, applying attention regularizations to both tasks gains more performance improvement than only to the ALSC task, which shows that our attention regularization approach can be extended to different tasks which involving aspect level attention weights, and works well in multi-task settings. For example, for the Binary classification in the Rest15 dataset, M-AT-LASTM outperforms AT-LSTM by 3.57% of accuracy and 4.96% of the F1 score, and M-CAN-2R_o further outperforms M-AT-LSTM by 3.28% of accuracy and 4.0% of the F1 score. 




Table <ref> shows the results of the ACD task in multi-task settings. Our proposed regularization terms can also improve the performance of ACD. Regularization R_o achieves the best performance in almost all metrics. 




 §.§ Attention Visualizations








Figure <ref> depicts the attention weights from AT-LSTM, M-AT-LSTM and M-CAN-2R_o methods, which are used to predict the sentiment polarity in the ALSC task. The subfigure (a), (b) and (c) show the attention weights of the same sentence, for the aspect food and service respectively. We observe that the attention weights of each word associated with each aspect are quite different for different methods. For AT-LSTM method in subfigure (a), the attention weights of aspect food and service are both high in words “outstanding”, “and”, and “the”, but actually, the word “outstanding” is used to describe the aspect food rather than service. The same situation occurs with the word “tops”, which should associate with
service rather than food. The attention mechanism alone is not good enough to locate aspect-specific opinion words and generate aspect-specific sentence representations in the ALSC task.
As shown in subfigure (b), the issue is mitigated in M-AT-LSTM. Multi-task learning can learn better hidden states of the sentence, and better aspect embeddings. However, it is still not good enough. For instance, the attention weights of the word “tops” are both high for the two aspects, and the weights are overlapped in the middle part of the sentence. 
As shown in subfigure (c), M-CAN+2R_o generates the best attention weights. The attention weights of the aspect food are almost orthogonal to the weights of service. The aspect food concentrates on the first part of the sentence while service on the second part. Meanwhile, the key opinion words “outstanding” and “tops” get highest attention weights in the corresponding aspects. 
We also visualize the attention for the auxiliary task ACD. Figure <ref> depicts the attention weights from the method M-CAN-2R_o. There are five predefined aspect categories (food, ambience, price, anecdotes/miscellaneous, service) in the dataset, two of which are mentioned in the sentence. In the ACD task, we need to calculate the attention weights for all the five aspect categories, and then generate aspect-specific sentence representations to determine whether the sentence contains each aspect. As shown in Figure <ref>, attention weights for aspects food and service are pretty good. The aspect food concentrates on words “food” and “outstanding”, and the aspect service focuses on the word “service”. It is interesting that for aspects which are not mentioned in the sentence, their attention weights often attend to meaningless stop words, such as “the”, “was”, etc. We do not distinguish these aspects and just treat them as a whole.
We plot the regularization loss curves in Figure <ref>, which shows that both R_s and R_o decrease during the training of AT-CAN-R_o.












 §.§ Case Studies

Overlapping Case We only add sparse regularization to overlapping sentences in which multiple aspects share the same opinion snippet. As shown in Figure <ref>, the sentence contains two aspects food and service, both described by the opinion snippet “highly disappointed”. Our method can locate the aspect terms and shared opinion words for both aspects, and then classify the sentiment correctly. 

Error Case With the help of attention visualization, we can conduct error analysis of our model conveniently. As shown in Figure <ref>, for the first sentence in error case, the aspect food attends on the right word “disappointing”, but fails to include the negation word “never”. This may be caused by the inaccurate sentence representation or aspect embedding. We can not rebuild the connection between the aspect and the word by our regularizations. The second sentence is negative but classified as neutral. The attention weights distribute evenly since the sentence does not contain any explicit opinion words. Since there is no tendency which words to concentrate, our model can not adjust the attention weights and help on such cases. 











§ CONCLUSIONS

We propose constrained attention networks for multi-aspect sentiment analysis, which handles multiple aspects of a sentence simultaneously. Specifically, we introduce orthogonal and sparse regularizations on attention weights. Furthermore, we introduce an auxiliary task ACD for promoting the ALSC task, and apply CAN on both tasks. Experimental results demonstrate that our approach outperforms the state-of-the-art methods. 



§ ACKNOWLEDGEMENT

This work is supported by National Science and Technology Major Project, China (Grant No. 2018YFB0204304).


acl_natbib


