title: How to train the Stanford NLP Sentiment Analysis tool
body: <p>Hell everyone! I'm using the Stanford Core NLP package and my goal is to perform sentiment analysis on a live-stream of tweets. </p>
  
  <p>Using the sentiment analysis tool as is returns a very poor analysis of text's 'attitude' .. many positives are labeled neutral, many negatives rated positive. I've gone ahead an acquired well over a million tweets in a text file, but I haven't a clue how to actually <em>train</em> the tool and create my own model.</p>
  
  <p><a href="http://nlp.stanford.edu/sentiment/code.html">Link to Stanford Sentiment Analysis page</a></p>
  
  <p>"Models can be retrained using the following command using the PTB format dataset:"</p>
  
  <pre><code>java -mx8g edu.stanford.nlp.sentiment.SentimentTraining -numHid 25 -trainPath train.txt -devPath     dev.txt -train -model model.ser.gz
  </code></pre>
  
  <p>Sample from dev.txt (The leading 4 represents polarity out of 5 ... 4/5 positive)</p>
  
  <pre><code>(4 (4 (2 A) (4 (3 (3 warm) (2 ,)) (3 funny))) (3 (2 ,) (3 (4 (4 engaging) (2 film)) (2 .))))
  </code></pre>
  
  <p>Sample from test.txt</p>
  
  <pre><code>(3 (3 (2 If) (3 (2 you) (3 (2 sometimes) (2 (2 like) (3 (2 to) (3 (3 (2 go) (2 (2 to) (2 (2 the) (2 movies)))) (3 (2 to) (3 (2 have) (4 fun))))))))) (2 (2 ,) (2 (2 Wasabi) (3 (3 (2 is) (2 (2 a) (2 (3 good) (2 (2 place) (2 (2 to) (2 start)))))) (2 .)))))
  </code></pre>
  
  <p>Sample from train.txt</p>
  
  <pre><code>(3 (2 (2 The) (2 Rock)) (4 (3 (2 is) (4 (2 destined) (2 (2 (2 (2 (2 to) (2 (2 be) (2 (2 the) (2 (2 21st) (2 (2 (2 Century) (2 's)) (2 (3 new) (2 (2 ``) (2 Conan)))))))) (2 '')) (2 and)) (3 (2 that) (3 (2 he) (3 (2 's) (3 (2 going) (3 (2 to) (4 (3 (2 make) (3 (3 (2 a) (3 splash)) (2 (2 even) (3 greater)))) (2 (2 than) (2 (2 (2 (2 (1 (2 Arnold) (2 Schwarzenegger)) (2 ,)) (2 (2 Jean-Claud) (2 (2 Van) (2 Damme)))) (2 or)) (2 (2 Steven) (2 Segal))))))))))))) (2 .)))
  </code></pre>
  
  <p>I have two questions going forward.</p>
  
  <p>What is the significance and difference between each file? Train.txt/Dev.txt/Test.txt ?</p>
  
  <p>How would I train my own model with a raw, unparsed text file full of tweets?</p>
  
  <p>I'm very new to NLP so if I am missing any required information or anything at all please critique! Thank you!</p>