title: Sentiment analysis using pyspark
body: <p>Since I am all new to <code>pyspark</code>, can anyone help me with the <code>pyspark</code> implementation of <em>sentiment analysis</em>. I have done the Python implementation. Can anyone tell me what changes are to be made?</p>
  
  <pre><code>import nltk
  import sys
  reload(sys)
  sys.setdefaultencoding("utf-8")
  from nltk.classify import NaiveBayesClassifier
  
  def format_sentence(sent):
    return({word: True for word in nltk.word_tokenize(sent)})
  
  #print(format_sentence("The cat is very cute"))
  
  pos = []
  with open("./pos_tweets.txt") as f:
  for i in f: 
      pos.append([format_sentence(i), 'pos'])
  
  neg = []
  with open("./neg_tweets.txt") as fp:
  for i in fp: 
      neg.append([format_sentence(i), 'neg'])
  
  # next, split labeled data into the training and test data
  training = pos[:int((.8)*len(pos))] + neg[:int((.8)*len(neg))]
  test = pos[int((.8)*len(pos)):] + neg[int((.8)*len(neg)):]
  
  classifier = NaiveBayesClassifier.train(training)
  
  example1 = "no!"
  
  print(classifier.classify(format_sentence(example1)))
  </code></pre>