title: What can multiprocessing and dill do together?
body: <p>I would like to use the <code>multiprocessing</code> library in Python.  Sadly <code>multiprocessing</code> uses <code>pickle</code> which doesn't support functions with closures, lambdas, or functions in <code>__main__</code>.  All three of these are important to me</p>
  
  <pre><code>In [1]: import pickle
  
  In [2]: pickle.dumps(lambda x: x)
  PicklingError: Can't pickle &lt;function &lt;lambda&gt; at 0x23c0e60&gt;: it's not found as __main__.&lt;lambda&gt;
  </code></pre>
  
  <p>Fortunately there is <a href="https://github.com/uqfoundation/dill" rel="noreferrer"><code>dill</code></a> a more robust pickle.  Apparently <code>dill</code> performs magic on import to make pickle work</p>
  
  <pre><code>In [3]: import dill
  
  In [4]: pickle.dumps(lambda x: x)
  Out[4]: "cdill.dill\n_load_type\np0\n(S'FunctionType'\np1 ...
  </code></pre>
  
  <p>This is very encouraging, particularly because I don't have access to the multiprocessing source code.  Sadly, I still can't get this very basic example to work</p>
  
  <pre><code>import multiprocessing as mp
  import dill
  
  p = mp.Pool(4)
  print p.map(lambda x: x**2, range(10))
  </code></pre>
  
  <p>Why is this?  What am I missing?  Exactly what are the limitations on the <code>multiprocessing</code>+<code>dill</code> combination?</p>
  
  <h2>Temporary Edit for J.F Sebastian</h2>
  
  <pre><code>mrockli@mrockli-notebook:~/workspace/toolz$ python testmp.py 
      Temporary Edit for J.F Sebastian
  
  mrockli@mrockli-notebook:~/workspace/toolz$ python testmp.py 
  Exception in thread Thread-2:
  Traceback (most recent call last):
    File "/home/mrockli/Software/anaconda/lib/python2.7/threading.py", line 808, in __bootstrap_inner
      self.run()
    File "/home/mrockli/Software/anaconda/lib/python2.7/threading.py", line 761, in run
      self.__target(*self.__args, **self.__kwargs)
    File "/home/mrockli/Software/anaconda/lib/python2.7/multiprocessing/pool.py", line 342, in _handle_tasks
      put(task)
  PicklingError: Can't pickle &lt;type 'function'&gt;: attribute lookup __builtin__.function failed
  
  ^C
  ...lots of junk...
  
  [DEBUG/MainProcess] cleaning up worker 3
  [DEBUG/MainProcess] cleaning up worker 2
  [DEBUG/MainProcess] cleaning up worker 1
  [DEBUG/MainProcess] cleaning up worker 0
  [DEBUG/MainProcess] added worker
  [DEBUG/MainProcess] added worker
  [INFO/PoolWorker-5] child process calling self.run()
  [INFO/PoolWorker-6] child process calling self.run()
  [DEBUG/MainProcess] added worker
  [INFO/PoolWorker-7] child process calling self.run()
  [DEBUG/MainProcess] added worker
  [INFO/PoolWorker-8] child process calling self.run()Exception in thread Thread-2:
  Traceback (most recent call last):
    File "/home/mrockli/Software/anaconda/lib/python2.7/threading.py", line 808, in __bootstrap_inner
      self.run()
    File "/home/mrockli/Software/anaconda/lib/python2.7/threading.py", line 761, in run
      self.__target(*self.__args, **self.__kwargs)
    File "/home/mrockli/Software/anaconda/lib/python2.7/multiprocessing/pool.py", line 342, in _handle_tasks
      put(task)
  PicklingError: Can't pickle &lt;type 'function'&gt;: attribute lookup __builtin__.function failed
  
  ^C
  ...lots of junk...
  
  [DEBUG/MainProcess] cleaning up worker 3
  [DEBUG/MainProcess] cleaning up worker 2
  [DEBUG/MainProcess] cleaning up worker 1
  [DEBUG/MainProcess] cleaning up worker 0
  [DEBUG/MainProcess] added worker
  [DEBUG/MainProcess] added worker
  [INFO/PoolWorker-5] child process calling self.run()
  [INFO/PoolWorker-6] child process calling self.run()
  [DEBUG/MainProcess] added worker
  [INFO/PoolWorker-7] child process calling self.run()
  [DEBUG/MainProcess] added worker
  [INFO/PoolWorker-8] child process calling self.run()
  </code></pre>