title: Restrictions on dynamically created functions with concurrent.futures.ProcessPoolExecutor
body: <p>I am trying to do some multiprocessing with functions that I dynamically create within other functions. It seems I can run these if the function fed to ProcessPoolExecutor is module-level:</p>
  
  <pre><code>def make_func(a):
      def dynamic_func(i):
          return i, i**2 + a
      return dynamic_func
  
  f_dyns = [make_func(a) for a in range(10)]
  def loopfunc(i):
      return f_dyns[i](i)
  
  with concurrent.futures.ProcessPoolExecutor(3) as executor:
      for i,r in executor.map(loopfunc, range(10)):
          print(i,":",r)
  </code></pre>
  
  <p>output:</p>
  
  <pre><code>0 : 0
  1 : 2
  2 : 6
  3 : 12
  4 : 20
  5 : 30
  6 : 42
  7 : 56
  8 : 72
  9 : 90
  </code></pre>
  
  <p>However I can't do it if the multiprocessing is launched by a class function:</p>
  
  <pre><code>class Test:
      def __init__(self,myfunc):
          self.f = myfunc
  
      def loopfunc(self,i):
          return self.f(i)
  
      def run(self):
          with concurrent.futures.ProcessPoolExecutor(3) as executor:
              for i,r in executor.map(self.loopfunc, range(10)):
                  print(i,":",r)
  
  o2 = Test(make_func(1))
  o2.run()
  </code></pre>
  
  <p>output:</p>
  
  <pre><code>Traceback (most recent call last):
    File "/home/farmer/anaconda3/envs/general/lib/python3.6/multiprocessing/queues.py", line 234, in _feed
      obj = _ForkingPickler.dumps(obj)
    File "/home/farmer/anaconda3/envs/general/lib/python3.6/multiprocessing/reduction.py", line 51, in dumps
      cls(buf, protocol).dump(obj)
  AttributeError: Can't pickle local object 'make_func.&lt;locals&gt;.dynamic_func'
  </code></pre>
  
  <p>On the other hand, I can run the multiprocessing on a class function just fine if I don't use a dynamically generated function in there. Is there some way around this? I tried adding the dynamically generated function to the 'globals' dictionary but that didn't seem to help:</p>
  
  <pre><code>def make_func_glob(a):
      def dynamic_func(i):
          return i, i**2 + a
      globals()['my_func_{0}'.format(a)] = dynamic_func
  
  make_func_glob(1)
  print("test:", my_func_1(3))
  o3 = Test(my_func_1)
  o3.run()
  </code></pre>
  
  <p>output:</p>
  
  <pre><code>test: (3, 10)
  Traceback (most recent call last):
    File "/home/farmer/anaconda3/envs/general/lib/python3.6/multiprocessing/queues.py", line 234, in _feed
      obj = _ForkingPickler.dumps(obj)
    File "/home/farmer/anaconda3/envs/general/lib/python3.6/multiprocessing/reduction.py", line 51, in dumps
      cls(buf, protocol).dump(obj)
  AttributeError: Can't pickle local object 'make_func_glob.&lt;locals&gt;.dynamic_func'
  </code></pre>
  
  <p>So python still thinks it is a local object even though I added it to the 'globals' dict. Something like this 'globals' idea would be fine, I don't need anything fancy. I am only dynamically creating these functions for the sake of convenience. I would be perfectly happy for them to be global objects. They will always be defined by the module, there are just a bunch of them with almost the same definition so it is more convenient to define them programmatically rather than writing them all out manually. So I would have thought it was possible to somehow get python to recognise them as "true" functions, like if I defined them via 'exec'. Or at least close enough that I could use them in my parallelised code.</p>