title: What is the breakdown of performance enhancement for vectorized UDFs?
body: <p>I'm looking at the vectorized UDFs in Spark 2.3.  In this <a href="https://www.youtube.com/watch?v=Til-StSDvfA" rel="nofollow noreferrer">this video</a>, it talks about (at least) two issues.</p>
  
  <ul>
  <li>Slow Serialization</li>
  <li>Inefficient data structures in python</li>
  </ul>
  
  <p>I completely understand the benefit of letting a data scientist work with the frameworks they are used to, and I see from a performance perspective in <a href="https://databricks.com/blog/2017/10/30/introducing-vectorized-udfs-for-pyspark.html" rel="nofollow noreferrer">this link</a>, this feature has a large benefit.  Does anyone have any data on how much of this is due to serialization improvement and how much is due to pandas?</p>