title: Spark's toDS vs to DF
body: <p>I understand that one can convert an RDD to a Dataset using <code>rdd.toDS</code>. However there also exists <code>rdd.toDF</code>.  Is there really any benefit of one over the other? </p>
  
  <p>After playing with the Dataset API for a day, I find out that almost any operation takes me out to a DataFrame (for instance <code>withColumn</code>). After converting an RDD with <code>toDS</code>, I often find out that another conversion to a DataSet is needed, because something brought me to a DataFrame again. </p>
  
  <p>Am I using the API wrongly? Should I stick with .toDF and only convert to a DataSet in the end of a chain of operations? Or is there a benefit to using toDS earlier?</p>
  
  <p>Here is a small concrete example</p>
  
  <pre><code>spark
    .read
    .schema (...)
    .json (...)
    .rdd
    .zipWithUniqueId
    .map[(Integer,String,Double)] { case (row,id) =&gt; ... }
    .toDS // now with a Dataset API (should use toDF here?)
    .withColumnRenamed ("_1", "id" ) // now back to a DataFrame, not type safe :(
    .withColumnRenamed ("_2", "text")
    .withColumnRenamed ("_2", "overall")
    .as[ParsedReview] // back to a Dataset
  </code></pre>