Spark Arrow, toPandas() and wide transformation
toPandas() takes your spark dataframe object and pulls all partitions on the client driver machine as a pandas dataframe.  Any operations on this new object (pandas dataframe) will be running on a single machine with python therefore no wide transformations will be possible because you aren't using spark cluster distributed computing anymore (i.e. no partitions/worker node interaction).
What does toPandas() actually do when using arrows optimization?
Is the resulting pandas dataframe safe for wide transformations (that requires data shuffling) on the pandas dataframe eg..merge operations? what about group and aggregate? What kind of performance limitation should I expect?
I am trying to standardize to Pandas dataframe where possible, due to ease of unit testing and swapability with in-memory objects without starting the monstrous spark instance.