Python: Non-Responsive multiprocessing.pool.map_async() function
Question: I have a strange problem here.

I have a python program that executes code held in seperate .py files,
designed to be executed in sequence, one after another. The codes work fine,
however they take too long to run. My plan was to split up processing each of
these .py files amongst 4 processors using
multiprocessing.pool.map_async(function, arguments) using execfile() as the
function and the filename as the argument.

So anyways, when I run the code, absolutely nothing happens at all, **not even
an error**.

Take a look and see if you can help me out, I run the file in
SeqFile.runner(SeqFile.file).

    
    
    class FileRunner:
        def __init__(self, file):
            self.file = file
        def runner(self, file):    
            self.run = pool.map_async(execfile, file)
    
    SeqFile = FileRunner("/Users/haysb/Dropbox/Stuart/Sample_proteins/Code/SVS_CodeParts/SequencePickler.py")
    VolFile = FileRunner("/Users/haysb/Dropbox/Stuart/Sample_proteins/Code/SVS_CodeParts/VolumePickler.py")
    CWFile = FileRunner("/Users/haysb/Dropbox/Stuart/Sample_proteins/Code/SVS_CodeParts/Combine_and_Write.py")
    (SeqFile.runner(SeqFile.file))
    


Answer: You have several problems here - I'm guessing you never used `multiprocessing`
before.

One of your problems is that you fire off an async operation but never wait
for it to end. If you _did_ wait for it to end, you'd get more info. For
example, add:

    
    
    result = SeqFile.run.get()
    

Do that, and you'll see the exception raised in the child process: you're
mapping `execfile` over the _string_ bound to `file`, so `execfile` sees one
character at a time. `execfile` barfs when the first thing it tries to do is
(in effect):

    
    
    execfile("/")
    

`apply_async()` would make a lot more sense, or `map_async()` passed a list of
all the files you want to run.

Etc - this gets tedious ;-)

## Specifics

Let's get rid of the irrelevant cruft here, and show a complete executable
program. I have three files `a.py`, `b.py` and `c.py`. Here's `a.py`:

    
    
    print "I'm A!"
    

The other two are the obvious variations.

Here's my entire driver:

    
    
    if __name__ == "__main__":
        import multiprocessing as mp
    
        files = ["a.py", "b.py", "c.py"]
        pool = mp.Pool(2)
        pool.imap_unordered(execfile, files)
        pool.close()
        pool.join()
    

That's all it takes, and prints (some permutation of):

    
    
    I'm A!
    I'm B!
    I'm C!
    

`imap_unordered()` splits the list of files up among the worker processes, and
doesn't care ("unordered") which order they run in. That's maximally
efficient. Note that I restricted the number of workers to 2, just to show
that it works fine even though there are more files (3) than worker processes
(2).

You can get any of the `Pool` functions to work similarly. If you _have_ ;-)
to use `map_async()`, for example, replace the `imap_unordered()` call with:

    
    
    async = pool.map_async(execfile, files)
    async.get()
    

Or:

    
    
    asyncs = [pool.apply_async(execfile, (fn,)) for fn in files]
    for a in asyncs:
        a.get()
    

Clearer? Keep it as simple as possible at first.

