title: Spark DataSet filter performance
body: <p>I have been experimenting different ways to filter a typed data set. It turns out the performance can be quite different. </p>
  
  <p>The data set was created based on a 1.6 GB rows of data with 33 columns and 4226047 rows. DataSet is created by loading csv data and mapped to a case class. </p>
  
  <pre><code>val df = spark.read.csv(csvFile).as[FireIncident]
  </code></pre>
  
  <p>A filter on UnitId = 'B02' should return 47980 rows. I tested three ways as below: 
  1) Use typed column (~ 500 ms on local host)</p>
  
  <pre><code>df.where($"UnitID" === "B02").count()
  </code></pre>
  
  <p>2) Use temp table and sql query (~ same as option 1)</p>
  
  <pre><code>df.createOrReplaceTempView("FireIncidentsSF")
  spark.sql("SELECT * FROM FireIncidentsSF WHERE UnitID='B02'").count()
  </code></pre>
  
  <p>3) Use strong typed class field (14,987ms, i.e. 30 times as slow)</p>
  
  <pre><code>df.filter(_.UnitID.orNull == "B02").count()
  </code></pre>
  
  <p>I tested it again with the python API, for the same data set, the timing is 17,046 ms, comparable to the performance of the scala API option 3.  </p>
  
  <pre><code>df.filter(df['UnitID'] == 'B02').count()
  </code></pre>
  
  <p>Could someone shed some light on how 3) and the python API are executed differently from the first two options? </p>