Precarious Popen Piping
Question: I want to use `subprocess.Popen` to run a process, with the following
requirements.

  1. I want to pipe the `stdout` and `stderr` back to the caller of `Popen` as the process runs.

  2. I want to kill the process after `timeout` seconds if it is still running.

I have come to the conclusion that a flaw in the `subprocess` API means it
cannot fulfill these two requirements at the same time. Consider the following
toy programs:

# chatty.py

    
    
    while True:
        print 'Hi'
    

# silence.py

    
    
    while True:
        pass
    

# caller.py

    
    
    import subprocess
    import time
    
    def go(command, timeout=60):
        proc = subprocess.Popen(command, shell=True,
                                stdout=subprocess.PIPE, stderr=subprocess.PIPE)
        start = time.time()
        while proc.poll() is None:
            print proc.stdout.read(1024) # <----- Line of interest
            if time.time() - start >= timeout:
                proc.kill()
                break
            else:
                time.sleep(1)
    

Consider the marked line above.

  * If it is included, `go('python silence.py')` will hang forever - not for just 60 seconds - because `read` is a blocking call until either `1024` bytes or end of stream, and neither ever comes.

  * If it is commented, `go('python chatty.py')` will be printing out `'Hi'` over and over, but how can it be streamed back as it is generated? `proc.communicate()` blocks until end of stream.

I would be happy with a solution that replaces requirement (1) above with "In
the case where a timeout did not occur, I want to get `stdout` and `stderr`
once the algorithm finishes." Even this has been problematic. My
implementation attempt is below.

# speech.py

    
    
    for i in xrange(0, 10000):
        print 'Hi'
    

# caller2.py

    
    
    import subprocess
    import time
    
    def go2(command, timeout=60):
        proc = subprocess.Popen(command, shell=True,
                                stdout=subprocess.PIPE, stderr=subprocess.PIPE)
        start = time.time()
        while True:
            if proc.poll() is not None:
                print proc.communicate()
                break
            elif time.time() - start >= timeout:
                proc.kill()
                break
            else:
                time.sleep(1)
    

But even this still has problems. Even though `python speech.py` runs in just
a couple seconds, `go2('python speech.py')` takes the full 60 seconds. This is
because the call to `print 'Hi'` in `speech.py` is blocking until
`proc.communicate()` is called when the process is killed. Since
`proc.stdout.read` had the problem demonstrated before with `silence.py`, I'm
really at a loss for how to get this working.

How can I get both the `stdout` and `stderr` and the timeout behavior?


Answer: The trick is to setup a side-band timer to kill the process. I wrote up a
program half way between chatty and silent:

    
    
    import time
    import sys
    
    for i in range(10,0,-1):
        print i
        time.sleep(1)
    

And then a program to kill it early:

    
    
    import subprocess as subp
    import threading
    import signal
    
    proc = subp.Popen(['python', 'longtime.py'], stdout=subp.PIPE, 
        stderr=subp.PIPE)
    timer = threading.Timer(3, lambda proc: proc.send_signal(signal.SIGINT),
        args=(proc,))
    timer.start()
    out, err = proc.communicate()
    timer.cancel()
    print proc.returncode
    print out
    print err
    

and it output:

    
    
    $ python killer.py
    1
    10
    9
    8
    
    Traceback (most recent call last):
      File "longtime.py", line 6, in <module>
        time.sleep(1)
    KeyboardInterrupt
    

Your timer could be made fancier, like trying increasingly bad signals til the
process completes, but you get the idea.

