Losing queued tasks in Python apply_async
Question: I am attempting to write a wrapper that iterates through running another
program with different input files. The program (over which I have no control,
but need to use) needs to be run out of the same directory as the input
file(s). So far my method is to use OS-module to change/create directory
structure, use apply-async to run the program given a sub-directory, and each
child in apply-async changes directory, creates the file, and the first 8
processes run successfully (i have 8 virtual cores)

However, I am queueing up to 100 of these processes (they run a simulation
which takes a few minutes, I'm looking to optimize). I use "call" on the
outside executable I am running. I thought everything was going great, but
then after the 8th simulation runs, everything stops, I check 0 processes are
running. It is as if the queue forgot about the other processes.

What can I do to fix this? I know my RAM only goes up about 300 MB out of 8GB.

Do I need to look into implementing some sort of queue myself that waits for
the exit code of the simulation executable?

Thank you in advance.


Answer: Maybe better than nothing. This shows a correct way to use `apply_async()`,
and demonstrates that - yup - there's no problem creating many more tasks than
processes. I'd tell you how this differs from what you're doing, but I have no
idea what you're doing ;-)

    
    
    import multiprocessing as mp
    
    def work(i):
        from time import sleep
        sleep(2.0 if i & 1 else 1.0)
        return i*i
    
    if __name__ == "__main__":
        pool = mp.Pool(4)
        results = [pool.apply_async(work, (i,)) for i in range(100)]
        results = [r.get() for r in results]
        print len(results), results
        pool.close()
        pool.join()
    

