title: Poor predictive performance for RandomForest in Spark
body: <p>This might be a long shot, but has anybody run into very poor predictive performance using RandomForest with Mllib? Here is what I'm doing: </p>
  
  <ul>
  <li>Spark 1.4.1 with PySpark </li>
  <li>Python 3.4.2 </li>
  <li>~30,000 Tweets of text </li>
  <li>12289 1s and 15956 0s </li>
  <li>Whitespace tokenization and then hashing trick for feature selection using 10,000 features </li>
  <li>Run RF with 100 trees and maxDepth of 4 and then predict using the features from all the 1s observations. </li>
  </ul>
  
  <p>So in theory, I should get predictions of close to 12289 1s (especially if the model overfits). But I'm getting exactly 0 1s, which sounds ludicrous to me and makes me suspect something is wrong with my code or I'm missing something. I notice similar behavior (although not as extreme) if I play around with the settings. But I'm getting normal behavior with other classifiers, so I don't think it's my setup that's the problem. </p>
  
  <p>For example: </p>
  
  <pre><code>&gt;&gt;&gt; lrm = LogisticRegressionWithSGD.train(lp, iterations=10) 
  &gt;&gt;&gt; logit_predict = lrm.predict(predict_feat) 
  &gt;&gt;&gt; logit_predict.sum() 
  9077 
  
  &gt;&gt;&gt; nb = NaiveBayes.train(lp) 
  &gt;&gt;&gt; nb_predict = nb.predict(predict_feat) 
  &gt;&gt;&gt; nb_predict.sum() 
  10287.0
  
  &gt;&gt;&gt; rf = RandomForest.trainClassifier(lp, numClasses=2, categoricalFeaturesInfo={}, numTrees=100, seed=422) 
  &gt;&gt;&gt; rf_predict = rf.predict(predict_feat) 
  &gt;&gt;&gt; rf_predict.sum() 
  0.0 
  </code></pre>
  
  <p>This code was all run back to back so I didn't change anything in between. Does anybody have a possible explanation for this? </p>