multiprocessing.pool context and load balancing
Question: I've encountered some unexpected behaviour of the python multiprocessing Pool
class.

Here are my questions:  
1) When does Pool creates its context, which is later used for serialization?  
_The example below runs fine as long as the Pool object is created after the
Container definition. If you swap the Pool initializations, serialization
error occurs. In my production code I would like to initialize Pool way before
defining the container class. Is it possible to refresh Pool "context" or to
achieve this in another way._  
2) Does Pool have its own load balancing mechanism and if so how does it work?  
_If I run a similar example on my i7 machine with the pool of 8 processes I
get the following results:  
\- For a light evaluation function Pool favours using only one process for
computation. It creates 8 processes as requested but for most of the time only
one is used (I printed the pid from inside and also see this in htop).  
\- For a heavy evaluation function the behaviour is as expected. It uses all 8
processes equally._  
3) When using Pool I always see 4 more processes that I requested (i.e. for
Pool(processes=2) I see 6 new processes). What is their role?

I use Linux with Python 2.7.2

    
    
    from multiprocessing import Pool
    from datetime import datetime
    
    POWER = 10
    
    def eval_power(container):
        for power in xrange(2, POWER):
            container.val **= power
        return container
    
    #processes = Pool(processes=2)
    
    class Container(object):
        def __init__(self, value):
            self.val = value
    
    processes = Pool(processes=2)
    
    if __name__ == "__main__":
        cont = [Container(foo) for foo in xrange(20)]
        then = datetime.now()
        processes.map(eval_power, cont)
        now = datetime.now()
        print "Eval time:", now - then
    

  
**EDIT - TO BAKURIU**  
1) I was afraid that that's the case.  
2) I don't understand what the linux scheduler has to do with python assigning
computations to processes. My situation can be ilustrated by the example
below:

    
    
    from multiprocessing import Pool
    from os import getpid
    from collections import Counter
    
    
    def light_func(ind):
        return getpid()
    
    
    def heavy_func(ind):
        for foo in xrange(1000000):
            ind += foo
        return getpid()
    
    
    if __name__ == "__main__":
        list_ = range(100)
        pool = Pool(4)
        l_func = pool.map(light_func, list_)
        h_func = pool.map(heavy_func, list_)
    
        print "light func:", Counter(l_func)
        print "heavy func:", Counter(h_func)
    

  
On my i5 machine (4 threads) I get the following results:  
_light func: Counter({2967: 100})  
heavy func: Counter({2969: 28, 2967: 28, 2968: 23, 2970: 21})_  
It seems that the situation is as I've described it. However I still don't
understand why python does it this way. My guess would be that it tries to
minimise communication expenses, but still the mechanism which it uses for
load balancing is unknown. The documentation isn't very helpful either, the
multiprocessing module is very poorly documented.  
3) If I run the above code I get 4 more processes as described before. The
screen comes from htop: <http://i.stack.imgur.com/PldmM.png>


Answer:   1. The `Pool` object creates the subprocesses during the call to `__init__` hence you **must** define `Container` before. By the way, I wouldn't include all the code in a single file but use a module to implement the `Container` and other utilities and write a small file that launches the main program.

  2. The `Pool` does exactly what is described in the [documentation](http://docs.python.org/2/library/multiprocessing.html#module-multiprocessing.pool). In particular it has no control over the scheduling of the processes hence what you see is what Linux's scheduler thinks it is right. For small computations they take so little time that the scheduler doesn't bother parallelizing them(this probably have better performances due to core affinity etc.)

  3. Could you show this with an example and what you see in the task manager? I think they may be the processes that handle the queue inside the `Pool`, but I'm not sure. On my machine I can see only the main process plus the two subprocesses.

* * *

Update on point 2:

The `Pool` object simply puts the tasks into a queue, and the child processes
get the arguments from this queue. If a process takes almost no time to
execute an object, than Linux scheduler let the process execute more
time(hence consuming more items from the queue). If the execution takes much
time then this scheduler will change processes and thus the other child
processes are also executed.

In your case a single process is consuming all items because the computation
take so little time that before the other child processes are ready it has
already finished all items.

As I said, `Pool` doesn't do anything about balancing the work of the
subprocesses. It's simply a queue and a bunch of workers, the pool puts items
in the queue and the processes get the items and compute the results. AFAIK
the only thing that it does to control the queue is putting a certain number
of tasks in a single item in the queue(see [the
documentation](http://docs.python.org/2/library/multiprocessing.html#multiprocessing.pool.multiprocessing.Pool.map))
but there is no guarantee about which process will grab which task. Everything
else is left to the OS.

On my machine the results are less extreme. Two processes get about twice the
number of calls than the other two for the light computation, while for the
heavy one all have more or less the same number of items processed. Probably
on different OSes and/or hardware we would obtain even different results.

