Pure Python / Python specific code in Pyspark application
The code you write (your "main" function) will be executed on the driver, and when you will operate on distributed data (e.g. RDDs and others), that executor will coordinate with the workers and handle that operation on them (in a sense, those operations will be executed on the workers).
I have recently got started with spark. I am using Python as the language for the spark application. What happens if we execute pure python code as a spark application (Spark). Does it will be executed on an executor or the driver itself.
Since the main function runs on spark, I think it should be on driver, but I have also read that spark does not do any computation, so will it run on executor then ? I may be missing something here. Would appreciate if anyone could explain this.