How to get around the pickling error of python multiprocessing without being in the top-level?
Question: I've researched this question multiple times, but haven't found a workaround
that either works in my case, or one that I understand, so please bear with
me.

Basically, I have a hierarchical organization of functions, and that is
preventing me from multiprocessing in the top-level. Unfortunately, I don't
believe I can change the layout of the program - because I need all the
variables that I create after the initial inputs.

For example, say I have this:

    
    
    import multiprocessing
    
      def calculate(x):
        # here is where I would take this input x (and maybe a couple more inputs)
        # and build a larger library of variables that I use further down the line
    
        def domath(y):
          return x * y
    
        pool = multiprocessing.Pool(3)
        final= pool.map(domath, range(3))
    
    calculate(2)
    

This yields the following error:

    
    
    Can't pickle <type 'function'>: attribute lookup __builtin__.function failed
    

I was thinking of globals, but I'm afraid that I'd have to define too many and
that may slow my program down quite a bit. Is there any workaround without
having to restructure the whole program?


Answer: The problem you encountered is actually a feature. The pickle source is
actually designed to prevent this sort of behavior in order to prevent
malicious code from being executed. Please consider that when addressing any
applicable security implementation.

First off we have some imports.

    
    
    import marshal
    import pickle
    import types
    

Here we have a function which takes in a function as an argument, pickles the
parts of the object, then returns a tuple containing all the parts:

    
    
    def pack(fn):
        code = marshal.dumps(fn.__code__)
        name = pickle.dumps(fn.__name__)
        defs = pickle.dumps(fn.__defaults__)
        clos = pickle.dumps(fn.__closure__)
        return (code, name, defs, clos)
    

Next we have a function which takes the four parts of our converted function.
It translates those four parts, and creates then returns a function out of
those parts. You should take note that globals are re-introduced into here
because our process does not handle those:

    
    
    def unpack(code, name, defs, clos):
        code = marshal.loads(code)
        glob = globals()
        name = pickle.loads(name)
        defs = pickle.loads(defs)
        clos = pickle.loads(clos)
        return types.FunctionType(code, glob, name, defs, clos)
    

Here we have a test function. Notice I put an import within the scope of the
function. Globals are not handled through our pickling process:

    
    
    def test_function(a, b):
        from random import randint
        return randint(a, b)
    

Finally we pack our test object and print the result to make sure everything
is working:

    
    
    packed = pack(test_function)
    print((packed))
    

Lastly, we unpack our function, assign it to a variable, call it, and print
its output:

    
    
    unpacked = unpack(*packed)
    print((unpacked(2, 20)))
    

Comment if you have any questions.

