multiprocessing : why are my processes not running in parallel?
Question: I am having a little bit of trouble understanding what is going on here. I
want to run some subprocess calls in parallel using the multiprocessing
module.

My simple example is basically calling a function that waits for 5 seconds,
prints an output then waits for another 5 seconds before terminating.

I want the function to be locked until it prints out a line of code (anything
really) and then allow other processes to continue working as well.

Here is an illustration:

    
    
    import time
    import subprocess
    import multiprocessing as mp
    
    def main(l):
    
        l.acquire()
        #Lock process while it is run
        proc = subprocess.Popen("python script_test.py", stdout=subprocess.PIPE)
        #Wait until one byte is read then release
        proc.stdout.read(1)
        l.release()
        #Terminate process when it has completed
        proc.poll()
    
    if __name__ == "__main__":
        #Initialize lockiing mechanism
        manager = mp.Manager()
        lock = manager.Lock()
    
        #Split up subprocess calls
        ncpu = mp.cpu_count()
        p = mp.Pool(ncpu)
        p.map(main, [lock]*ncpu)
        print "Run time = {0}s using {1} processes on {2}.".format(time.time()-start, ncpu, ncpu)
    

Where the contents of `script_test.py` is as follows:

    
    
    import time
    #Do some work up until the point of the print statement
    time.sleep(5)
    
    print "Okay 5 seconds has passed!"
    #Continue doing work while allowing other processes to proceed
    time.sleep(5)
    

It seems like this is not running in parallel. The total time takes 40.7s for
me with `ncpu` equal to 4, but there should only be about `5*ncpu + 5*2 = 25`
plus overhead since the first 5 seconds are locked and the last 5 should not
for each process.

I am running the `script_test.py` as an example, but in practice this will be
an executable file.

My questions are:

  1. How can I utilize locks properly to accomplish the above?
  2. I am running the script from a command prompt but I don't see any output. How would I print the output of the subprocess from within the child processes launched by the multiprocessing module?


Answer:   1. Because you have a single shared lock being used incorrectly, you've turned your code into a serial execution. What happened is that one of your processes got the lock, and the other three are waiting on that first process to execute. Once the first process is done, the next one grabs the lock and runs the script while the other two wait (and so on and so forth). So, you're creating four processes, but you're only running them one at time the way your are using your lock. From your code, I don't think you really need the lock anyway. The processes are not really sharing data, because each subprocess.Popen call is it's own instance.

  2. The subprocess spins off yet another process to execute your script, and you'll need to use subprocess methods to read the "stdout" of the script and then print that out again. Use calls like "check_output" instead of calling Popen for example, save them to a variable and print.

EDIT - Question 1 has changed - new answer:

  1. You have to look at your current problem at a more fundamental level. If I'm understanding your comments correctly, it sounds like your processes are operating on the same text file that is both modified and read. You can't operate on it in parallel (at least not very well), since you'll constantly be locking around it. If you can make separate text files like separate output names of some sort, then you can start to make things operate in parallel. I don't know what your executable is doing exactly, so it will be hard to say. You'll need to also call your lock acquisition in separate areas around the modify and the read sections of your code, so that may entail you importing this test script into your multiprocessing script. If the modify and write action is bound up in the executable (and can't create separate text files), then you're going to have a tough time with this problem.

