Memory usage keep growing with Python's multiprocessing.pool
Question: Here's the program:

    
    
    #!/usr/bin/python
    
    import multiprocessing
    
    def dummy_func(r):
        pass
    
    def worker():
        pass
    
    if __name__ == '__main__':
        pool = multiprocessing.Pool(processes=16)
        for index in range(0,100000):
            pool.apply_async(worker, callback=dummy_func)
    
        # clean up
        pool.close()
        pool.join()
    

I found memory usage (both VIRT and RES) kept growing up till close()/join(),
is there any solution to get rid of this? I tried maxtasksperchild with 2.7
but it didn't help either.

I have a more complicated program that calles apply_async() ~6M times, and at
~1.5M point I've already got 6G+ RES, to avoid all other factors, I simplified
the program to above version.

**EDIT:**

Turned out this version works better, thanks for everyone's input:

    
    
    #!/usr/bin/python
    
    import multiprocessing
    
    ready_list = []
    def dummy_func(index):
        global ready_list
        ready_list.append(index)
    
    def worker(index):
        return index
    
    if __name__ == '__main__':
        pool = multiprocessing.Pool(processes=16)
        result = {}
        for index in range(0,1000000):
            result[index] = (pool.apply_async(worker, (index,), callback=dummy_func))
            for ready in ready_list:
                result[ready].wait()
                del result[ready]
            ready_list = []
    
        # clean up
        pool.close()
        pool.join()
    

I didn't put any lock there as I believe main process is single threaded
(callback is more or less like a event-driven thing per docs I read).

I changed v1's index range to 1,000,000, same as v2 and did some tests - it's
weird to me v2 is even ~10% faster than v1 (33s vs 37s), maybe v1 was doing
too many internal list maintenance jobs. v2 is definitely a winner on memory
usage, it never went over 300M (VIRT) and 50M (RES), while v1 used to be
370M/120M, the best was 330M/85M. All numbers were just 3~4 times testing,
reference only.


Answer: Use `map_async` instead of `apply_async` to avoid excessive memory usage.

For your first example, change the following two lines:

    
    
    for index in range(0,100000):
        pool.apply_async(worker, callback=dummy_func)
    

to

    
    
    pool.map_async(worker, range(100000), callback=dummy_func)
    

It will finish in a blink before you can see its memory usage in `top`. Change
the list to a bigger one to see the difference. But note `map_async` will
first convert the iterable you pass to it to a list to calculate its length if
it doesn't have `__len__` method. If you have an iterator of a huge number of
elements, you can use `itertools.islice` to process them in smaller chunks.

I had a memory problem in a real-life program with much more data and finally
found the culprit was `apply_async`.

P.S., in respect of memory usage, your two examples have no obvious
difference.

