title: PySpark how is pickle used in SparkSql and Dataframes
body: <p>I am trying to understand how PySpark uses pickle for RDDs and avoids it for SparkSql and Dataframes. The basis of the question is from slide#30 in <a href="https://www.slideshare.net/hkarau/a-really-really-fast-introduction-to-py-spark-lightning-fast-cluster-computing-with-python-1" rel="nofollow noreferrer">this</a> link.I am quoting it below for reference:</p>
  
  <p>"[PySpark] RDDs are generally RDDs of pickled objects. Spark SQL (and DataFrames) avoid some of this". </p>
  
  <p>How is pickle used in Spark Sql?</p>