title: Could Pandas ever run on Spark?
body: <p>We have a Spark cluster that is intended for performing analysis of various financial data sets in Python, ranging up to hundreds of terabytes. It works well for those comfortable with Spark but recently we've had a number of data scientists who are more familiar with Pandas requesting to use Pandas instead of Spark. The main advantage of Spark is that it is designed for distributed processing and can handle larger data sets, so we've been looking in to setting up a Dask cluster.</p>
  
  <p>But it got me thinking: is there some conceptual reason why Pandas scripts could not run on a Spark cluster by implementing a compatibility layer to translate Pandas primitives into Spark primitives, the same way Dask works? Or are they fundamentally different in some way?</p>