Spark: Dangers of using Python
In your driver application, you don't necessarily have to collect a ton of records. Maybe you're just doing a reduce down to some statistics.
This is just typical behavior: Drivers usually deal with statistical results. Your mileage may vary.
On the other hand, Spark applications typically use the executors to read in as much data as their memory allows and process it. So memory management is almost always a concern.
I think this is the distinction the book is getting at.
In the book "Spark: The definitive guide" (currently early release, text might change), the authors advise against the use of Pyspark for user-defined functions in Spark:
"Starting up this Python process is expensive but the real cost is in serializing the data to Python. This is costly for two reasons, it is an expensive computation but also once the data enters Python, Spark cannot manage the memory of the worker. This means that you could potentially cause a worker to fail if it becomes resource constrained (because both the JVM and python are competing for memory on the same machine)."
I understand that the competition for worker node resources between Python and the JVM can be a serious problem. But doesn't that also apply to the driver? In this case, it would be an argument against using Pyspark at all. Could anyone please explain what makes the situation different on the driver?