Embarassingly parallel tasks with IPython Parallel (or other package) depending on unpickable objects
Question: I often hit problems where I wanna do a simple stuff over a set of many, many
objects quickly. My natural choice is to use IPython Parallel for its
simplicity, but often I have to deal with unpickable objects. After trying for
a few hours I usually resign myself to running my taks overnight on a single
computer, or do a stupid thing like dividing things semi-manually in to run in
multiple python scripts.

To give a concrete example, suppose I want to delete all keys in a give S3
bucket.

What I'd normally do without thinking is:

    
    
    import boto
    from IPython.parallel import Client
    
    connection = boto.connect_s3(awskey, awssec)
    bucket = connection.get_bucket('mybucket')
    
    client = Client()
    loadbalancer = c.load_balanced_view()
    
    keyList = list(bucket.list())
    loadbalancer.map(lambda key: key.delete(), keyList)
    

The problem is that the `Key` object in `boto` is unpickable (*). This occurs
very often in different contexts for me. It's a problem also with
multiprocessing, execnet, and all other frameworks and libs I tried (for
obvious reasons: they all use the same pickler to serialize the objects).

Do you guys also have those problems? Is there a way I can serialize these
more complex objects? Do I have to write my own pickler for this particular
objects? If I do, how do I tell IPython Parallel to use it? How do I write a
pickler?

Thanks!

* * *

(*) I'm aware that I can simply make a list of the keys names and do something
like this:

    
    
    loadbalancer.map(lambda keyname: getKey(keyname).delete())
    

and define the `getKey` function in each engine of the IPython cluster. This
is just a particular instance of a more general problem that I find often.
Maybe it's a bad example, since it can be easily solved in another way.


Answer: `IPython` has a `use_dill` option, where if you have the `dill` serializer
installed, you can serialize most "unpicklable" objects.

[How can I use dill instead of pickle with
load_balanced_view](http://stackoverflow.com/questions/24042443/how-can-i-use-
dill-instead-of-pickle-with-load-balanced-view)

