Python itertools with multiprocessing - huge list vs inefficient CPUs usage with iterator
Question: I work on n elements (named "pair" below) variations with repetition used as
my function's argument. Obviously everything works fine as long as the "r"
list is not big enough to consume all the memory. The issue is I have to make
more then 16 repetitions for 6 elements eventually. I use 40 cores system in
cloud for this.

The code looks looks like the following:

    
    
    if __name__ == '__main__':
      pool = Pool(39)
      r = itertools.product(pairs,repeat=16)
      pool.map(f, r)
    

I believe i should use iterator instead of creating the huge list upfront and
here the problem starts..

I tried to solve the issue with the following code:

    
    
    if __name__ == '__main__':
      pool = Pool(39)
      for r in itertools.product(pairs,repeat=14):
        pool.map(f, r)
    

The memory problem goes away but the CPUs usage is like 5% per core. Now the
single core version of the code is faster then this.

I'd really appreciate if you could guide me a bit..

Thanks.


Answer: Your original code isn't creating a `list` upfront in your own code
(`itertools.product` returns a generator), but `pool.map` is realizing the
whole generator (because it assumes if you can store all outputs, you can
store all inputs too).

Don't use `pool.map` here. If you need ordered results, using `pool.imap`, or
if result order is unimportant, use `pool.imap_unordered`. Iterate the result
of either call (don't wrap in `list`), and process the results as they come,
and memory should not be an issue:

    
    
    if __name__ == '__main__':
        pool = Pool(39)
        for result in pool.imap(f, itertools.product(pairs, repeat=16)):
            print(result)
    

If you're using `pool.map` for side-effects, so you just need to run it to
completion but the results and ordering don't matter, you could dramatically
improve performance by using `imap_unordered` and using `collections.deque` to
efficiently drain the "results" without actually storing anything (a `deque`
with `maxlen` of `0` is the fastest, lowest memory way to force an iterator to
run to completion without storing the results):

    
    
    from collections import deque
    
    if __name__ == '__main__':
        pool = Pool(39)
        deque(pool.imap_unordered(f, itertools.product(pairs, repeat=16)), 0)
    

Lastly, I'm a little suspicious of specifying 39 `Pool` workers;
`multiprocessing` is largely beneficial for CPU bound tasks; if you're using
using more workers than you have CPU cores and gaining a benefit, it's
possible `multiprocessing` is costing you more in IPC than it gains, and using
more workers is just masking the problem by buffering more data.

If your work is largely I/O bound, you might try using a thread based pool,
which will avoid the overhead of pickling and unpickling, as well as the cost
of IPC between parent and child processes. Unlike process based pools, Python
threading is subject to
[GIL](https://wiki.python.org/moin/GlobalInterpreterLock) issues, so your CPU
bound work in Python (excluding GIL releasing calls for I/O, `ctypes` calls
into .dll/.so files, and certain third party extensions like `numpy` that
release the GIL for heavy CPU work) is limited to a single core (and in Python
2.x for CPU bound work you often waste a decent amount of that resolving GIL
contention and performing context switches; Python 3 removes most of the
waste). But if your work is largely I/O bound, blocking on I/O releases the
GIL to allow other threads to run, so you can have many threads as long as
most of them delay on I/O. It's easy to switch too (as long as you haven't
designed your program to rely on separate address spaces for each worker by
assuming you can write to "shared" state and not affect other workers or the
parent process), just change:

    
    
    from multiprocessing import Pool
    

to:

    
    
    from multiprocessing.dummy import Pool
    

and you get the
[`multiprocessing.dummy`](https://docs.python.org/3/library/multiprocessing.html#module-
multiprocessing.dummy) version of the pool, based on threads instead of
processes.

