Building a Sentiment Corpus of Tweets in Brazilian Portuguese
Henrico Bertini Brum Maria das Graças Volpe Nunes
cs.CL






§ INTRODUCTION
 

Sentiment Analysis (SA) became a popular area of Natural Language Processing in the last decade. The classification of semantic orientation of documents is a challenge for artificial intelligence methods since it is based not only on the regular meaning of words, but also on their semantic role in the context and on the author's intention. Furthermore, the amount of data available in blogs, social media posts and forums has created a great opportunity for researchers to build datasets for evaluating methods and studying new linguistic phenomena.

Websites on e-commerce, movie reviews and hotel reservations usually allow the user to an objective evaluation besides the written commentaries. This objective evaluation (binary recommendation, star score, 10-point scale) can be a good feature for automatic labeling large datasets on semantic orientation, thus improving the resources for researches over the past decades <cit.>.

The limitation of this technique is the data available in this conditions. Social media, for example, is a large source of user opinions and evaluation <cit.>, but the lack of an objective score attached to the posts demands a manual annotation in order to data become useful for SA, even though the data is enriched by linguistic phenomena such as expressions, slangs and irony. 

Manual annotation ends up being more expensive and time consuming, since it demands several guarantees of accuracy, such as developing guidelines, training annotators and revising the annotation <cit.>.

In this paper we introduce TweetSentBR (TTsBR), a corpus manually annotated with data extracted from Twitter. The section <ref> presents some related work on SA and corpus annotation. Section <ref> presents the corpus and its properties, such as the size, the annotation tags, the information on annotators and the process of data extraction. Section <ref> presents some data analysis and polarity classification experiments on the corpus. Section <ref> is a brief discussion on the importance of the corpus and how it can be used in Brazilian Portuguese research on SA.





§ RELATED WORK
 
Several works present new methods and approaches for tasks such as polarity classification <cit.>, detection of irony <cit.> and aspect extraction in text <cit.>.

One of the major issues of this area is the building of datasets for evaluating methods and for training machine learning models. turney2002thumbs, one of the first works on polarity classification, used product reviews labeled as “recommended” and “not recommended”. The source of the data was a website called Epinions, where users could evaluate products and leave a five star score for each review. The authors considered any review with less than 3 stars as “not recommended”. pang2002thumbs uses a similar score (star rating) in order to compile a corpus of movie reviews on three classes (positive, negative and neutral).

The automatic approach worked very well for building large datasets, but the method limited research on domains where users input an objective score. Despite of the challenges of the manual annotation, researches began building new datasets by training annotators to label the data. socher2013recursive introduces Stanford Sentiment Treebank, a re-labeling of the previous IMDB corpus presented in <cit.>. SemEval, an important semantic evaluation event, also produces several datasets for English designed for SA tasks <cit.>. Some authors even used distant supervision techniques for automatic labeling large datasets quickly using features such as emoticons <cit.>.

In Brazilian Portuguese, several works presented corpora for SA. freitas2012vampiro introduce ReLi, a sentiment corpus of book reviews manually annotated in three classes (positive, neutral and negative). The authors have chosen books from different publics in order to vary the linguistic phenomena in the corpus (from teenage books to literature classics). ReLi contains annotation of semantic orientation, part-of-speech tagging and aspect of opinion, and it was later used as resource for researches in SA <cit.>. One of the issues on this corpus observed on the literature is the unbalanced classes - the majority of sentences is neutral (72%), while the negative class represents only 4% of the data.

On the product review domain, hartmann2014large presented Buscape corpus, a large corpora in Brazilian Portuguese. The corpus contains 13.685 reviews labeled as positive and negative, using scores given by users on Buscape, a popular e-commerce website. A similar dataset is Mercado Livre corpus, introduced in avancco2015normalizaccao, containing 43.818 product reviews also labeled automatically and balanced between the two classes.

silva2011effective collected a corpus from Twitter in Portuguese. The dataset was collected by searching two entities in the social network (Dilma and Serra, two running candidates at the time) and manually annotated as positive or negative. The corpus contains 76.358 documents balanced between positive and negative. The corpus was originally constructed for sentiment stream analysis meaning it contains several retweets and links, phenomena that may interfere on sentiment classification but is vital to maintain the stream for the former task.

Also on binary polarity classification, moraes20157x1 introduce the Corpus 7x1, a brazilian portuguese corpus on Twitter comments during the 2014 World Cup semi-finals. The corpus presents some interesting user behavior such as irony, sarcasm, cheering and angry due to the final match score. Corpus 7x1 contains 2.728 tweets labeled manually in three classes - the neutral class represents tweets that do not align with wither positive or negative sentiments.

moraes2016classificaccao also uses Twitter as the source of data, but compile a corpus of computer products containing 2.317 tweets. The data is manually labeled in three classes and the authors also performed experiments on SA using lexical-based classifiers and SVM.

A large Twitter corpus was compiled by <cit.> using distant supervision. The authors labeled tweets in Brazilian Portuguese using emojis representing positive and negative sentiments following the work of go2009twitter in English. The corpus contains 554.623 positive tweets and 425.444 negative. The approach is a fast way to label data, but the method can not guarantee the absence of noise data such as irony, sarcasm or incorrect labels.




§ TWEETSENTBR
 
TweetSentBr is composed of 15.000 tweets (17.166 tokens) extracted using Python-Twitter [<https://github.com/bear/python-twitter>], a wrapper for Twitter API. Due to the limitations of Twitter API, we developed a continuous crawler in order to obtain documents during the first semester of 2017. The final dataset is split in two documents - a training set with 12.999 documents labeled in positive (44%), neutral (26%) and negative (29%); and a test set composed of 2001 documents with similar distribution to the training set, 45%, 25% and 29% respectively. See <ref> for the number of documents in each class.





 §.§ Data source

Data was extracted from Twitter between January and July in 2017. We chose to focus on the TV show domain because of the large amount of user generated content on Twitter during the exhibition of the shows. Hashtags (#) are used on social media to group messages on topics and the TV shows usually ask for its audience to use a specific hashtag in order to get visibility in these social networks. Some of the program hashtags group hundreds of thousands of messages during the exhibition of a show and that content can represent suggestions, complaints, evaluations and questions to the entities related to the programs.

We empirically defined nine programs from three major TV channels in Brazil based on its popularity and activeness in social media. Talk-shows, reality shows (gastronomy and music) and variety shows were chosen in order to diversify the phenomena in the corpus. The periodicity of the exhibitions are also different, some shows go live daily when others go live once or twice a week.

Since we were looking for user generated content, we ignored documents generated by public entities, such as celebrities, companies, TV channels or any official user on Twitter. We also discarded retweets, which are the reposts of popular posts in the social network.



 §.§ Classes definition
 

Following hovy2010towards recommendations, a codebook or manual was written to ensure the agreement between annotators. The codebook contains examples, definitions and tips for the annotation process. The rules and guidelines were formed by empirically observing the dataset crawled before the annotation update based on the feedback from the annotators during the early stages of annotation.

The definitions were created based on the domain and the input received by the annotators after the first contact with the data. These are the guidelines for the annotation in TTsBR:

Positive class: Positive sentences describe feelings of pleasure, satisfaction, compliment or recommendation. The target of the sentiment must be the TV show or any entity related to it (host, guests, audience, sketches, invited bands...). Positive comparisons, such as “This show is better than the other” are considered positive and emojis can be strong indicatives of positivity.



Negative class: Negative sentences describe feelings of disagreement, disapprove, complaint or hate. The target of the sentiment must be the TV show or any entity related to it (hosts, guests, audience, sketches, invited bands...). Negative sentences can be direct, as in “Today's show is terrible...” or implied in a suggestion, as in “the host could improve its jokes, right?”. Factual information such as delays, abrupt cuts or technical failures are also considered negative as long as it refers for the show or any entity related to it. Emojis are also good indicatives of negativity.



Neutral class: The neutral label must be used for any sentence the annotator could not identify as an opinion (positive or negative) direct or implied. Factual sentences that do not represent a hit or a flaw, such as “Show X just started”, inaccurate semantic orientation (“Don't know what to think about this”) and sentences the annotator can not completely comprehend were instructed to be annotated as neutral. Some tweets in the corpus were generated by social media robots (most of them on audience measurements) and the annotators labeled these as neutral as well.



We also wanted to keep track of the sentences that most caused doubt in the annotators. The annotators had a check box to mark in case of doubt in the annotation, even though this option did not prevent the annotator from labeling the sentiment of the sentence. The annotators were instructed to mark the doubt option every time they felt divided between two or more classes or when they took more than the average time (2 minutes) in a sentence.
The addition of a doubt option gives us new information on the data and also reduces the stress on annotators. In the first stages of annotation, only an average of 10% of sentences were marked.





 §.§ Annotation process

For the annotation process we recruited seven native speakers of Brazilian Portuguese in three different areas - linguistics, journalism and computer science. The annotation process was based on hovy2010towards, following the eight steps of annotation in order to improve the reliability of the resource.

We developed a user friendly interface for the annotators to label the tweets (<ref>). The interface contains the codebook, the phases of annotation, a progress bar and a panel with tweets for labeling. The annotation panel shows to the annotator the three classes and a box to be checked when in doubt (even though every tweet must have a label chosen in order to proceed to the next phase) and a side box with quick tips, contact information and a link to the codebook.




Each annotator received a set with one hundred tweets to be labeled. After this step we measured the average time, agreement (all received the same set) and we took notes of the questions about the codebook guidelines. Then we proceeded to rewrite the codebook, adding more examples and detailing the definitions based on the questions presented on the first annotation.

The next step was a meeting with all annotators to receive the feedback of the annotators, when the tweets were revised by everyone and we presented the new version of the codebook.

We then proceeded to the regular annotation. First the participants labeled 300 tweets in order to measure agreement. We used Krippendorf's Alpha <cit.> to measure the agreement of the annotators. In this phase we obtained 52.9% on the nominal measure and 70% on the interval measure. The annotators began the individual phases when each one labeled around 2.000 tweets in six weeks, completing the training portion of the corpus.

Two supervisors annotated a small portion of the corpus in order to obtain a test set revised. The goal was to form a 10% part of the dataset specially labeled for evaluating machine learning methods. Some of the data annotated in the agreement phase was also used to compose this set.

For the release, we define the general sentiment for each document based on a major voting of the labels provided by each annotator. Some documents were only labeled by one annotator, while other were annotated by 3 or 7 annotators. 45 documents tied and have no sentiment label, even though they were kept in the dataset with a “none” label.



 §.§ Release and distribution
 
The dataset is available in <http://bitbucket.org/HBrum/tweetsentbr/>. Twitter has a Privacy Policy forbidding the redistribution of the data, so we managed to provide only the ids of the tweets in the corpus. Any user with a working identification can search for the tweets freely.

We provide the dataset with the ids, the hashtags used in the search, full annotators labels, the count of how many annotators checked the doubt option and the general sentiment for each document, as well as a tool for downloading the dataset as long as having a valid credential (provided by Twitter itself). An example of the dataset is presented below:






§ EXPERIMENTS
 

In order to investigate the properties of the corpora, we defined a series of experiments to determine word frequency, the class balance, and we also performed some polarity classification using baseline methods for Portuguese.

For the experiments, we performed a preprocessing of the data - we replaced numbers (dates, currency values) by a NUMBER token, we also replaced user names and links by the tokens USERNAME and URL, respectively. We trimmed repetition of characters (eg. “looooove” turns into “love”) to a minimum of 3 repeat characters



 §.§ Corpus statistics


In order to extract some information from the corpus, we measured the relevance of words on each class. We calculated the tf-idf value of each term ignoring hashtags, stop-words, emojis and punctuation. We chose to report only the polarity classes (positive and negative) since the neutral class groups several characteristics (facts, out-of-topic sentences, confusing content) that led the analysis to terms not expressive, such as the name of the shows, users nicknames and neutral verbs (present, watch,...).

The terms indicated in <ref> show a notable semantic orientation represented in the classes - the positive class shows the verb “to love” and positive adjectives, while the negative class shows adjectives (the word “trash” is popular used as adjective on Twitter) and the verb “to remove” that may indicate a request for removing a guest from a show or even a participant from a reality show.





 §.§ Polarity classification task


In order to evaluate the corpus on the polarity classification task we used three machine learning methods proposed in avancco2015normalizaccao in product review corpora for Brazilian Portuguese. We ran two classic classifiers, Naive Bayes and SVM (linear kernel), and a hybrid approach initially proposed for cross-domain polarity classification.

The hybrid classifier combines SVM and a lexical-based approach for binary sentiment classification. The lexical-based classifier uses linguistic rules  designed for binary classification, such as identifying strong sentiment words in sentences and increasing or decreasing their value based on the presence of terms (eg. very, less, much). The lexical-based method is used when the sentence representation is located near the hyperplane (in a threshold assumed as 0.5 by the former author). The work has been used in other works in the literature <cit.>.

We represented the sentences in vector using a binary bag-of-words (presence/absence of terms), sentiment words (obtained through Portuguese lexicons), emoticons and POS tags. The methods were built using Scikit-learn library <cit.>.

<ref> presents the results obtained for the binary classification which is a common approach in the area <cit.>. The neutral class was not used in this experiments.




For the three class classification we followed the pipeline approach used in moraes2016classificaccao - first we classify in objective and subjective (meaning neutral or polarity sentences, respectively) and then classify the polarity one in positive or negative. We did not evaluate the corpora using the Hybrid Classifier since it uses a lexical-based approach that could not be re-factored to analyze the neutral class. The results are shown in <ref>






§ DISCUSSION AND FUTURE WORK
 

TweetSentBR is a manually annotated corpus designed for polarity classification. The corpus was formed using a novel domain for the Brazilian Portuguese language what can be exploited by new machine learning approaches such as deep learning architectures and ensembles.

It also offers new resources for linguistic approaches on natural language by observing the expressions, social media behavior or hate speech detection. The doubt label, for example, can be used for a better evaluation of classifiers by comparing machine learning flaws with human uncertainty on labeling data.

This corpora also differs from other approaches by including the neutral class. The addition of the neutral class approximates the corpora to popular applications, since the polarity classifiers available in the industry must find solutions for separating the opinions of users from noisy data. ReLi <cit.> and Computer-BR <cit.> are the only corpora we found in the literature that describes the use of a neutral class on the annotation.

We believe this corpora can still be improved by labeling more data manually or by using semi-supervised methods, such as self-training or co-training <cit.>.



§ ACKNOWLEDGEMENT
 

We acknowledge financial support from CNPq and the help of Amanda Carneiro, Fernando Nóbrega, Juliana Batista, Rafael Anchiêta and Thales Bertaglia. for the volunteer annotation of TTsBR and also Marcos Treviso for the development of the annotation interface.




§ BIBLIOGRAPHICAL REFERENCES



lrec


