How to implement custom control over python multiprocessing.Pool?
Question: Usually i use following code, and it works fine when you do not matter in
which order function `process_func` will handle some parameter:

    
    
    params = [1,2,3,4,5 ... ]
    
    def process_func():
        ...
    
    pool = new Pool(40)
    pool.map(process_func, params)
    pool.close()
    pool.join()
    

In example above we have processes of one type, with maximum simultanious
number of 40. But.. imagine we have processes (parameters) of different type,
which should be executed simultaniously. For example, in my selenium grid i
have 40 firefox, 40 chromes. And i have 5000 test cases, some of them prefer
chrome, some of them firefox, some of them does not matter.

For example, lets say we have following types:

  * type firefox: maximum simultanious number: 40
  * type chrome: maximum simultanious number: 40

In this case our pool will have maximum of 80 simultanious processes, but
there is strict rule: 40 of them must be firefox, 40 of them must be chromes.

It means that params won't be taken one after each other. Pool must select
value from params list in a way to have maximum of each process type.

How it is possible to achieve that?


Answer: I would modify your `process_func()` to take one more parameter that tells it
which "type" to be and use two separate pools. Adding
[functools.partial](https://docs.python.org/2/library/functools.html#functools.partial)
will allow us to still use `pool.map()`:

    
    
    from functools import partial
    from multiprocessing import Pool
    
    params = [1,2,3,4,5 ... ]
    
    def process_func(type, param):
        if type == 'Firefox':
            # do Firefox stuff
        else:
            # do Chrome stuff
    
    chrome_pool = Pool(40)
    fox_pool = Pool(40)
    
    chrome_function = partial(process_func, 'Chrome')
    fox_function = partial(process_func, 'Firefox')
    
    chrome_pool.map(chrome_func, params)
    fox_pool.map(fox_func, params)
    
    chrome_pool.close()
    fox_pool.close()
    chrome_pool.join()
    fox_pool.join()
    

The `functools.partial()` function allows us to bind an argument to a specific
value, by returning a new function object that will always supply that
argument. This approach allows you to limit each "type" (for lack of a better
term) to 40 worker processes.

