Lock process until subprocess reaches a certain point
Question: I am running multiple subprocesses in parallel, but I need to lock each
process until the subprocess gives an output (via print function). The
subprocesses are running a python script that has been packaged to an
executable.

The code looks like this:

    
    
    import multiprocessing as mp
    import subprocess
    import os
    
    
    def main(args):
    
        l,inpath = args
        l.acquire()
        print "Running KNN.exe for files in %s" %   os.path.normpath(inpath).split('\\')[-1] 
    
        #Run KNN executable as a subprocess
        subprocess.call(os.path.join(os.getcwd(), "KNN.exe"))
        #This is where I want to wait for any output from the subprocess before releasing the lock
        l.release()
    
        #Here I would like to wait until subprocess is done then print that it is done
        l.acquire()
        print "Done %s" % os.path.normpath(inpath).split('\\')[-1]
        l.release()
    
    
    if __name__ == "__main__":
        #Set working directory path containing input text file
        os.chdir("C:\Users\Patrick\Google Drive\KNN")
        #Get folder names in directory containing GCM input
        manager = mp.Manager()
        l = manager.Lock()
        gcm_dir = "F:\FIDS_GCM_Data_CMIP5\UTRB\UTRB KNN-CAD\Input"
        paths = [(l, os.path.join(gcm_dir, folder)) for folder in os.listdir(gcm_dir)]    
        #Set up multiprocessing pool
        p = mp.Pool(mp.cpu_count())
        #Map function through input paths
        p.map(main, paths)
    

So the goal is to lock the process so that a subprocess can be run until
receiving an output. After which the lock can be released and the subprocess
can continue, until it is complete, then I'd like to print that it is
complete.

My question is how can I wait for the single (and only) output from the
subprocess before releasing the lock on the process (out of multiple)?

Additionally how can I wait for the process to terminate then print that it is
complete?


Answer: Your code makes use of the `call` method, which already waits for the
subprocess to finish (which means all output has already been generated). I'm
inferring from your question you'd like to be able to differentiate between
when output is first written and when the subprocess is finished. Below is
your code with my recommended modifications inline:

    
    
    def main(args):
    
        l,inpath = args
        l.acquire()
        print "Running KNN.exe for files in %s" %   os.path.normpath(inpath).split('\\')[-1]
    
        #Run KNN executable as a subprocess
        #Use the Popen constructor
        proc = subprocess.Popen(os.path.join(os.getcwd(), "KNN.exe"), stdout=subprocess.PIPE)
    
        #This is where I want to wait for any output from the subprocess before releasing the lock
        # Wait until the subprocess has written at least 1 byte to STDOUT (modify if you need different logic)
        proc.stdout.read(1)
    
        l.release()
    
        #Here I would like to wait until subprocess is done then print that it is done
        #proc.wait()
        (proc_output, proc_error) = proc.communicate()
    
        l.acquire()
        print "Done %s" % os.path.normpath(inpath).split('\\')[-1]
        l.release()
    

Note that the above doesn't assume you want to do anything with the
subprocess's output other than check that it has been generated. If you want
to do anything with that output that is less trivial than the above (consume 1
byte then drop it on the floor), the `proc.stdout` (which is a `file` object)
should represent everything that the subprocess generates while running.

