Passing objects to Spark
If your object's aren't picklable your options are pretty limited. If you can create them on the executor side though (frequently a useful option for things like database connections), you can parallelize a  regular list (e.g. maybe a list of the constructor parameters) and then use map if your dostuff function returns (picklable) values you want to use or foreach if your dostuff function is called for its side effects (like updating a database or similar).
I try to understand the capabilities of Spark, but I fail to see if the following is possible in Python.
I have some objects that are non Pickable (wrapped from C++ with SWIG).
I have a list of those objects obj_list = [obj1, obj2, ...]
All those objects have a member function called .dostuff
I'd like to parallelized the following loop in Spark (in order to run it on AWS since I don't have a big architecture internally. We could probably use multiprocessing, but I don't think I can easily send the objects over the network):
[x.dostuff() for x in obj_list]
Any pointers would be appreciated.