How to synchronize the output of Python subprocess
Question: I think I'm having issues to synchronize the output of two Popen running
concurrently. It seems that the output from these two different command lines
are interleaved with one another. I also tried using RLock to prevent this
from happening but it didn't work.

A sample output would be:

    
    
    cmd1
    cmd1
    cmd2
    cmd2
    cmd2
    cmd2
    cmd1
    cmd2
    

The code is as attached:

    
    
    import subprocess
    import threading
    
    class PopenWorkerThread(threading.Thread):
        def __init__(self, cmdLine):
            self.lock = threading.RLock()
            self.WebSphereCmdLine = cmdLine
            threading.Thread.__init__(self)
    
        def run(self):
            logger.error('Runninf: ' + self.WebSphereCmdLine)
            proc = subprocess.Popen(self.WebSphereCmdLine, shell=False, stdout=subprocess.PIPE)
            while True:
                self.lock.acquire()
                print proc.stdout.readline()
                self.lock.release()
    
    def launch():
        commandLine = ['ls -l', 'netstat']
        for cmdLine in commandLine:
            workerThread = PopenWorkerThread(cmdLine)
            workerThread.start()
    
    launch()
    

Is there a way to synchronize the outputs so that they look like such?

    
    
    cmd1
    cmd1
    cmd1
    cmd1
    cmd1
    cmd2
    cmd2
    cmd2
    cmd2
    cmd2
    


Answer: You're locking with a granularity of a line, so of course lines from one
thread can and do alternate with lines from the other. As long as you're
willing to wait until a process ends before showing any of its output, you can
lock with the larger "process" granularity. Of course you have to use the SAME
lock for both threads -- having each thread use a completely separate lock, as
you're doing now, cannot achieve anything at all, obviously.

So, for example:

import subprocess import threading

class PopenWorkerThread(threading.Thread):

    
    
    lock = threading.RLock()  # per-class, NOT per-instance!
    
    def __init__(self, cmdLine):
        self.WebSphereCmdLine = cmdLine
        threading.Thread.__init__(self)
    
    def run(self):
        logger.error('Runninf: ' + self.WebSphereCmdLine)
        proc = subprocess.Popen(self.WebSphereCmdLine, shell=False, stdout=subprocess.PIPE)
        result, _ = proc.communicate()
        with self.lock:
            print result,
    

def launch(): commandLine = ['ls -l', 'netstat'] for cmdLine in commandLine:
workerThread = PopenWorkerThread(cmdLine) workerThread.start()

launch()

