Shared pool map between processes with object-oriented python
Question: (**python2.7**)

I'm trying to do a kind of scanner, that has to walk through CFG nodes, and
split in different processes on branching for parallelism purpose.

The scanner is represented by an object of class Scanner. This class has one
method **traverse** that walks through the said graph and splits if necessary.

Here how it looks:

    
    
    class Scanner(object):
        def __init__(self, atrb1, ...):
           self.attribute1 = atrb1
           self.process_pool = Pool(processes=4)
        def traverse(self, ...):
            [...]
            if branch:
               self.process_pool.map(my_func, todo_list).
    

My problem is the following: How do I create a instance of
multiprocessing.Pool, that is shared between all of my processes ? I want it
to be shared, because since a path can be splitted again, I do not want to end
with a kind of fork bomb, and having the same Pool will help me to limit the
number of processes running at the same time.

The above code does not work, since Pool can not be pickled. In consequence, I
have tried that:

    
    
    class Scanner(object):
       def __getstate__(self):
          self_dict  = self.__dict__.copy()
          def self_dict['process_pool']
          return self_dict
        [...]
    

But obviously, it results in having **self.process_pool** not defined in the
created processes.

Then, I tried to create a Pool as a module attribute:

    
    
    process_pool = Pool(processes=4)
    
    def my_func(x):
        [...]
    
    class Scanner(object):
        def __init__(self, atrb1, ...):
           self.attribute1 = atrb1
        def traverse(self, ...):
            [...]
            if branch:
               process_pool.map(my_func, todo_list)
    

It does not work, and this
[answer](http://stackoverflow.com/questions/2782961/yet-another-confusion-
with-multiprocessing-error-module-object-has-no-attribu) explains why. But
here comes the thing, wherever I create my Pool, something is missing. If I
create this Pool at the end of my file, it does not see self.attribute1, the
same way it did not see
[answer](http://stackoverflow.com/questions/2782961/yet-another-confusion-
with-multiprocessing-error-module-object-has-no-attribu) and fails with an
AttributeError.

I'm not even trying to share it yet, and I'm already stuck with
Multiprocessing way of doing thing.

I don't know if I have not been thinking correctly the whole thing, but I can
not believe it's so complicated to handle something as simple as "having a
worker pool and giving them tasks".

Thank you,

**EDIT** : I resolved my first problem (AttributeError), my class had a
callback as its attribute, and this callback was defined in the main script
file, after the import of the scanner module... But the concurrency and "do
not fork bomb" thing is still a problem.


Answer: What you want to do can't be done safely. Think about if you somehow had a
single shared `Pool` shared across parent and worker processes, with, say, two
worker processes. The parent runs a `map` that tries to perform two tasks, and
each task needs to `map` two more tasks. The two parent dispatched tasks go to
each worker, and the parent blocks. Each worker sends two more tasks to the
shared pool and blocks for them to complete. But now all workers are now
occupied, waiting for a worker to become free; you've deadlocked.

A safer approach would be to have the workers return enough information to
dispatch additional tasks in the parent. Then you could do something like:

    
    
    class MoreWork(object):
        def __init__(self, func, *args):
            self.func = func
            self.args = args
    
    pool = multiprocessing.Pool()
    try:
        base_task = somefunc, someargs
        outstanding = collections.deque([pool.apply_async(*base_task)])
        while outstanding:
            result = outstanding.popleft().get()
            if isinstance(result, MoreWork):
                outstanding.append(pool.apply_async(result.func, result.args))
            else:
                ... do something with a "final" result, maybe breaking the loop ...
    finally:
         pool.terminate()
    

What the functions are is up to you, they'd just return information in a
`MoreWork` when there was more to do, not launch a task directly. The point is
to ensure that by having the parent be solely responsible for task dispatch,
and the workers solely responsible for task completion, you can't deadlock due
to all workers being blocked waiting for tasks that are in the queue, but not
being processed.

This is also not at all optimized; ideally, you wouldn't block waiting on the
first item in the queue if other items in the queue were complete; it's a lot
easier to do this with the `concurrent.futures` module, specifically with
[`concurrent.futures.wait`](https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.wait)
to wait on the first available result from an arbitrary number of outstanding
tasks, but you'd need a third party PyPI package to get `concurrent.futures`
on Python 2.7.

