title: Random Forest in Spark
body: <p>So I am trying to classify certain text documents into three classes. </p>
  
  <p>I wrote the following code for Cross validation in spark</p>
  
  <pre><code>from pyspark.ml.tuning import CrossValidator, ParamGridBuilder
  from pyspark.ml.evaluation import MulticlassClassificationEvaluator
  # Define a grid of hyperparameters to test:
  #  - maxDepth: max depth of each decision tree in the GBT ensemble
  #  - maxIter: iterations, i.e., number of trees in each GBT ensemble
  # In this example notebook, we keep these values small.  In practice, to get the highest accuracy, you would likely want to try deeper trees (10 or higher) and more trees in the ensemble (&gt;100).
  
  paramGrid = ParamGridBuilder()\
    .addGrid(jpsa.rf.maxDepth, [2,4,10])\
    .addGrid(jpsa.rf.numTrees, [100, 250, 600,800,1000])\
    .build()
  
  # We define an evaluation metric.  This tells CrossValidator how well we are doing by comparing the true labels with predictions.
  evaluator = MulticlassClassificationEvaluator(metricName="f1", labelCol=jpsa.rf.getLabelCol(), predictionCol=jpsa.rf.getPredictionCol())
  # Declare the CrossValidator, which runs model tuning for us.
  cv = CrossValidator(estimator=pipeline, evaluator=evaluator, estimatorParamMaps=paramGrid,numFolds=5)
  
  cvModel=cv.fit(jpsa.jpsa_train)
  
  evaluator.evaluate(cvModel.transform(jpsa.jpsa_train))
  </code></pre>
  
  <p>I dont have much data. 115 total observation(documents with labels). I break them into 80:35 training and test. On the training I use 5 fold cross validation using the above code. </p>
  
  <p>The evaluator above gave me the following on whole training data. </p>
  
  <pre><code>evaluator.evaluate(cvModel.transform(jpsa.jpsa_train))
  0.9021290600237969
  </code></pre>
  
  <p>I am using f1 here since I am not able to find aucROC for MulticlassEvaluator in Spark as the option for evaluator. It does have it for Binary. I know AUC is for binary class, but then we can get a combined or average AUC for multi class by plotting various binary classes and getting their AUC. Sri-kit learn does the same for Multi class AUC. </p>
  
  <p>However, when I use the evaluator on test data, my f1 score is miserable. </p>
  
  <pre><code>evaluator.evaluate(cvModel.transform(jpsa.jpsa_test))
  0.5830903790087463
  </code></pre>
  
  <p>This indicates it's overfitting. Also if I don't use 1000 and 800 trees in the hyparameter search space and just keep it to 600 and 800, my test accuracy is 69%. So that means more trees are leading to overfitting? Which is strange as that is contrary to what and how Random Forests work. More tress reduce variance and lead to less overfitting (in fact ppl even suggest sometimes random forests don't overfit though I disagree that with very less data and complex forest it can). </p>
  
  <p>Is that what is happening here? Less data and more no. of trees leading to overfitting? </p>
  
  <p>Also how do I get a measure of cross validation accuracy? Currently evaluator is on training data. I don't want that as a measure to choose the algo. I want the validation data. Is it possible to get this OOB estimate internally from this CV estimator?</p>