how to classify tweets in python based on handful of domains I am currently working on a project that classifies tweets based on certain categories of information they belong to (7 given categories for his project). 

For example, a tweet with the keywords "I think NY should ban smoking" should be classified as a tweet in the "Pollution" category with negative sentiment.

I have been able to get the sentiment analysis to work somewhat, but need some help with getting the category part to work. I am also open to all solutions.

My code so far is the stream.py module. And the following command to get live twitter data into a text file: python stream.py &gt; output.txt:

import oauth2 as oauth
import urllib2 as urllib

api_key = 'xx'
api_secret = 'xx'

access_token_key = 'x-x'
access_token_secret = 'x'

_debug = 0

oauth_token    = oauth.Token(key=access_token_key, secret=access_token_secret)
oauth_consumer = oauth.Consumer(key=api_key, secret=api_secret)

signature_method_hmac_sha1 = oauth.SignatureMethod_HMAC_SHA1()

http_method = "GET"


http_handler  = urllib.HTTPHandler(debuglevel=_debug)
https_handler = urllib.HTTPSHandler(debuglevel=_debug)

'''
Construct, sign, and open a twitter request
using the hard-coded credentials above.
'''
def twitterreq(url, method, parameters):
  req = oauth.Request.from_consumer_and_token(oauth_consumer,
                                             token=oauth_token,
                                             http_method=http_method,
                                             http_url=url, 
                                             parameters=parameters)

  req.sign_request(signature_method_hmac_sha1, oauth_consumer, oauth_token)

  headers = req.to_header()

  if http_method == "POST":
    encoded_post_data = req.to_postdata()
  else:
    encoded_post_data = None
    url = req.to_url()

  opener = urllib.OpenerDirector()
  opener.add_handler(http_handler)
  opener.add_handler(https_handler)

  response = opener.open(url, encoded_post_data)

  return response
#locations=-74,40,-73,41
def fetchsamples():
  url = "https://stream.twitter.com/1.1/statuses/filter.json?track=money&amp;locations=-74,40,-73,41"
  parameters = []
  response = twitterreq(url, "POST", parameters)
  for line in response:
    print(line.strip())

if __name__ == '__main__':
  fetchsamples()


The sentiment of a tweet is computed as the sum of the sentiment scores for each term in the tweet. Run: python tweet_sentiment.py AFINN-111.txt tweet_file to get the tweet sentiments.

Here is the link for my upload for AFINN-111.txt.http://s000.tinyupload.com/index.php?file_id=62473255612293859764

Here is the code for tweet_sentiment.py

import sys
import json
import ast
import re

def calcScoreFromTerm(termScoreFile):   # returns a dictionary with term-score values
    scores ={}
    for line in termScoreFile:
        term, score = line.split("\t")
        scores[term] = float(score)
    return scores

def getTweetText(tweet_file):   #returns a list of all tweets
    tweets = []
    for line in tweet_file:
        # print line
        jsondata = json.loads(line)
        if "text" in jsondata.keys():
            tweets.append(jsondata["text"])
    tweet_file.close()
    return tweets

def filterTweet(et):
    # Remove punctuations and non-alphanumeric chars from each tweet string
    pattern = re.compile('[^A-Za-z0-9]+')
    et = pattern.sub(' ', et)
    #print encoded_tweet

    words = et.split()

    # Filter unnecessary words
    for w in words:
        if w.startswith("RT") or w.startswith("www") or w.startswith("http"):
            words.remove(w)

    return words
I don't see your sentiment analysis code in your tweet_sentiment.py, so I will not comment on that.

And if I understand correctly, you are trying to build a classifier for text classification based on Twitter data.

So, depending on your data size, here are two things you could try.


Small data size (several tens of thousands or lower): Tdidf + Support Vector Machine, basically convert your text to a numerical vector form representing the existence of certain words along with their importance relative to the text and the whole dataset. Then feed the data into an SVM classifier, lots of tutorials are out there, and here is a good one.
Big data size (hundreds of thousands and more)(and ready to have fun): (word embedding Or one-hot encoding) + (CNN Or RNN)
, word embedding could represent the semantic similarity with others into a vector space model and have a dense representation, a good explanation could be found here. As for using CNN to do text classification you could refer to this and this Text understanding from scratch. and another one for RNN