nowwe can load our imdb review datasetsubset out the last , reviews which will be used for our analysisand normalize them using the following snippet in [ ]dataset pd read_csv( 'movie_reviews csv'reviews np array(dataset['review']sentiments np array(dataset['sentiment']extract data for model evaluation test_reviews reviews[ :test_sentiments sentiments[ :sample_review_ids [ normalize dataset norm_test_reviews tn normalize_corpus(test_reviewswe also extract out some sample reviews so that we can run our models on them and interpret their results in detail bing liu' lexicon this lexicon contains over , words which have been divided into two files named positive-words txtcontaining around , words/phrases and negative-words txtwhich contains , words/phrases the lexicon has been developed and curated by bing liu over several years and has also been explained in detail in his original paper by nitin jindal and bing liu"identifying comparative sentences in text documentsproceedings of the th annual international acm sigirseattle if you want to use this lexiconyou can get it from which also includes link to download it as an archive (rar formatmpqa subjectivity lexicon the term mpqa stands for multi-perspective question answering and it contains diverse set of resources pertaining to opinion corporasubjectivity lexiconsubjectivity sense annotationsargument lexicondebate corporaopinion finderand many more this is developed and maintained by the university of pittsburgh and their official web site subjectivity lexicon is part of their opinion finder framework and contains subjectivity clues and contextual polarity details on this can be found in the paper by theresa wilsonjanyce wiebeand paul hoffmann"recognizing contextual polarity in phrase-level sentiment analysisproceeding of hlt-emnlp- you can download the subjectivity lexicon from their official web site at lexicons/subj_lexicon/contains subjectivity clues present in the dataset named subjclueslen hltemnlp tff the following snippet shows some sample lines from the lexicon type=weaksubj len= word =abandonment pos =noun stemmed = priorpolarity=negative type=weaksubj len= word =abandon pos =verb stemmed = priorpolarity=negative type=strongsubj len= word =zenith pos =noun stemmed = priorpolarity=positive type=strongsubj len= word =zest pos =noun stemmed = priorpolarity=positive each line consists of specific word and its associated polaritypos tag informationlength (right now only words of length are present)subjective contextand stem information