title: How to evaluate a classifier with PySpark 2.4.5
body: <p>I'm wondering what the best way is to evaluate a fitted binary classification model using Apache Spark 2.4.5 and PySpark (Python). I want to consider different metrics such as accuracy, precision, recall, auc and f1 score.</p>
  
  <p>Let us assume that the following is given:</p>
  
  <pre><code># pyspark.sql.dataframe.DataFrame in VectorAssembler format containing two columns: target and features
  # DataFrame we want to evaluate
  df
  
  # Fitted pyspark.ml.tuning.TrainValidationSplitModel (any arbitrary ml algorithm)
  model
  </code></pre>
  
  <p><strong>1. Option</strong></p>
  
  <p>Neither <a href="https://spark.apache.org/docs/2.4.5/api/scala/index.html#org.apache.spark.ml.evaluation.BinaryClassificationEvaluator" rel="noreferrer">BinaryClassificationEvaluator</a> nor <a href="https://spark.apache.org/docs/2.4.5/api/scala/index.html#org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator" rel="noreferrer">MulticlassClassificationEvaluator</a> can calculate all metrics mentioned above on their own. Thus, we use both evaluators.</p>
  
  <pre><code>from pyspark.ml.evaluation import BinaryClassificationEvaluator, MulticlassClassificationEvaluator
  
  # Create both evaluators
  evaluatorMulti = MulticlassClassificationEvaluator(labelCol="target", predictionCol="prediction")
  evaluator = BinaryClassificationEvaluator(labelCol="target", rawPredictionCol="prediction", metricName='areaUnderROC')
  
  # Make predicitons
  predictionAndTarget = model.transform(df).select("target", "prediction")
  
  # Get metrics
  acc = evaluatorMulti.evaluate(predictionAndTarget, {evaluatorMulti.metricName: "accuracy"})
  f1 = evaluatorMulti.evaluate(predictionAndTarget, {evaluatorMulti.metricName: "f1"})
  weightedPrecision = evaluatorMulti.evaluate(predictionAndTarget, {evaluatorMulti.metricName: "weightedPrecision"})
  weightedRecall = evaluatorMulti.evaluate(predictionAndTarget, {evaluatorMulti.metricName: "weightedRecall"})
  auc = evaluator.evaluate(predictionAndTarget)
  </code></pre>
  
  <p><em>Downside</em></p>
  
  <ul>
  <li>It seems weird and contradictory to use MulticlassClassificationEvaluator when evaluating a binary classifier</li>
  <li>I have to use two different evaluators to calculate five metrics</li>
  <li>MulticlassClassificationEvaluator only calculates <code>weightedPrecision</code> and <code>weightedRecall</code> (which is ok for a multi class classification). However, are these two metrics equal to <code>precision</code> and <code>recall</code> in a binary case ?</li>
  </ul>
  
  <p><strong>2. Option</strong></p>
  
  <p>Using RDD based API with <a href="https://spark.apache.org/docs/2.4.5/api/scala/index.html#org.apache.spark.mllib.evaluation.BinaryClassificationMetrics" rel="noreferrer">BinaryClassificatinMetrics</a> and <a href="https://spark.apache.org/docs/2.4.5/api/scala/index.html#org.apache.spark.mllib.evaluation.MulticlassMetrics" rel="noreferrer">MulticlassMetrics</a>. Again, both metrics can't calculate all metrics mentioned above on their own (at least not in python ..). Thus, we use both.</p>
  
  <pre><code>from pyspark.mllib.evaluation import BinaryClassificationMetrics, MulticlassMetrics
  
  # Make prediction
  predictionAndTarget = model.transform(df).select("target", "prediction")
  
  # Create both evaluators
  metrics_binary = BinaryClassificationMetrics(predictionAndTarget.rdd.map(tuple))
  metrics_multi = MulticlassMetrics(predictionAndTarget.rdd.map(tuple))
  
  acc = metrics_multi.accuracy
  f1 = metrics_multi.fMeasure(1.0)
  precision = metrics_multi.precision(1.0)
  recall = metrics_multi.recall(1.0)
  auc = metrics_binary.areaUnderROC
  </code></pre>
  
  <p><em>Downsides</em></p>
  
  <ul>
  <li>According to Spark, RDD-based API is now in maintenance mode and <a href="https://spark.apache.org/docs/2.4.5/ml-guide.html" rel="noreferrer">DataFrame-based API is primary API</a> </li>
  <li>Again, I have to use two different metrics to calculate five metrics</li>
  <li>Again, using MulticlassMetrics seems contradictory when evaluating a binary classifier</li>
  </ul>
  
  <p><em>Upside</em></p>
  
  <ul>
  <li>In my case (~1.000.000 rows) Option 2 seems to be faster than Option 1 </li>
  </ul>
  
  <p><em>Surprise</em></p>
  
  <ul>
  <li>In my case I get different <code>f1</code> and <code>areaUnderRoc</code> values when using Option 1 vs when using Option 2.</li>
  </ul>
  
  <p><strong>Option 3</strong></p>
  
  <p>Use numpy and sklearn</p>
  
  <pre><code>import numpy as np
  from sklearn.metrics import accuracy_score, precision_score, recall_score, roc_auc_score, f1_score
  
  # Make predicitons
  predictionAndTarget = model.transform(df).select("target", "prediction")
  
  predictionAndTargetNumpy = np.array((predictionAndTarget.collect()))
  
  acc = accuracy_score(predictionAndTargetNumpy[:,0], predictionAndTargetNumpy[:,1])
  f1 = f1_score(predictionAndTargetNumpy[:,0], predictionAndTargetNumpy[:,1])
  precision = precision_score(predictionAndTargetNumpy[:,0], predictionAndTargetNumpy[:,1])
  recall = recall_score(predictionAndTargetNumpy[:,0], predictionAndTargetNumpy[:,1])
  auc = roc_auc_score(predictionAndTargetNumpy[:,0], predictionAndTargetNumpy[:,1])
  </code></pre>
  
  <p><em>Downside</em></p>
  
  <ul>
  <li>Using sklearn and numpy seems to be weird as Apache Spark is claiming to have there own API for Evaluation</li>
  <li>Using numpy and sklearn can even be impossible if the dataset is getting too big. </li>
  </ul>
  
  <p>To summarize my questions:</p>
  
  <ol>
  <li>Which Option above (if any) is recommended for evaluating a binary classifier using Apache Spark 2.4.5 and PySpark.</li>
  <li>Are there other Options? Am I missing something important?</li>
  <li>Why do I get different results for the metrics when using Option 1 vs when using Option 2</li>
  </ol>