title: What is the difference between Spark DataSet and RDD
body: <p>I'm still struggling to understand the full power of the recently introduced Spark Datasets.</p>
  
  <p>Are there best practices of when to use RDDs and when to use Datasets?</p>
  
  <p>In their <a href="https://databricks.com/blog/2016/01/04/introducing-spark-datasets.html">announcement</a> Databricks explains that by using Datasets staggering reductions in both runtime and memory can be achieved. Still it is claimed that Datasets are designed ''to work alongside the existing RDD API''. </p>
  
  <p>Is this just a reference to downward compatibility or are there scenarios where one would prefer to use RDDs over Datasets?</p>