"\bwouldn' \ ""would not" "\bcan' \ ""can not" "\bcannot\ ""can not"def create_ngram_model(params=none)def preprocessor(tweet)global emoticons_replaced tweet tweet lower(#return tweet lower(for in emo_repl_ordertweet tweet replace(kemo_repl[ ]for rrepl in re_repl iteritems()tweet re sub(rrepltweetreturn tweet tfidf_ngrams tfidfvectorizer(preprocessor=preprocessoranalyzer="word"certainlythere are many more abbreviations that could be used here but already with this limited setwe get an improvement for sentiment versus not sentiment of half pointwhich comes to percent=pos vs neg = =pos/neg vs irrelevant/neutral = =pos vs rest = =neg vs rest = taking the word types into account so far our hope was to simply use the words independent of each other with the hope that bag-of-words approach would suffice just from our intuitionhoweverneutral tweets probably contain higher fraction of nounswhile positive or negative tweets are more colorfulrequiring more adjectives and verbs what if we could use this linguistic information of the tweets as wellif we could find out how many words in tweet were nounsverbsadjectivesand so onthe classifier could maybe take that into account as well