Reading stderr of subprocess while it is executing
Question: I'd like to read what is written to stderr by a subprocess while it is
executing.

However, when I use this script that I've written, stderr does not seem to
have anything for me to read until the subprocess has exited.

    
    
    #!/usr/bin/env python2
    
    import sys
    from subprocess import Popen, PIPE, STDOUT
    
    if len(sys.argv) < 2:
        print "Please provide a command"
        sys.exit(1)
    
    sub = Popen(sys.argv[1:], stdout=PIPE, stderr=STDOUT)
    
    for i, line in enumerate(sub.stdout):
        sys.stdout.write("%d: %s" % (i, line))
    

**edit:**

Ok, I've gotten closer now. If I specify the number of bytes to read it
overcomes the buffering.

    
    
    #!/usr/bin/env python2
    
    import sys 
    from subprocess import Popen, PIPE, STDOUT
    
    if len(sys.argv) < 2:
        print "Please provide a command"
        sys.exit(1)
    
    sub = Popen(sys.argv[1:], stdout=PIPE, stderr=STDOUT)
    
    i = 0 
    while sub.poll() is None:
        line = sub.stdout.read(64)
        line.strip("\b")
        sys.stdout.write("%d: %s\n" % (i, line))
        i += 1
    

Snippet of output:

    
    
    58:  86 q=21.0 size= 4541841kB time=00:00:22.08 bitrate=1685014.2kbi
    frame=  567 fps= 86 q=22.0 size= 4543667kB time=00:00:2
    frame=  621 fps= 87 q=20.0 sizs/s    
    frame= 4545352kB time=00:00:26.11 bitrate=1425939.2kbits/s    
    62: =  686 fps= 90 q=12.0 size= 4546970kB time=00:00:28.89 bitrate=1
    frame=  758 fps= 93 q=25.0 size= 4548534kB t
    frame=  794 fps= 92 bitrate=1168185.5kbits/s    
    65:  q=27.0 size= 4550901kB time=00:00:33.40 bitrate=1115897.0kbits/
    frame=  827 fps= 91 q=27.0 size= 4552324kB time=00:00:34.7
    frame=  857 fps= 89 q=26.0 size=  
    frame=  254kB time=00:00:36.12 bitrate=1032874.9kbits/s    
    69: 892 fps= 88 q=25.0 size= 4556598kB time=00:00:37.36 bitrate=9988
    frame=  948 fps= 89 q=19.0 size= 4558565kB time=
    frame= 1006 fps= 90 q=19937320.4kbits/s    
    72: .0 size= 4560139kB time=00:00:42.16 bitrate=885880.0kbits/s    
    73: frame= 1060 fps= 91 q=19.0 size= 4561958kB time=00:00:44.49 bitr
    frame= 1122 fps= 93 q=18.0 size= 4563460
    frame= 1173 fps=0:47.08 bitrate=793898.4kbits/s
    

It looks like my problem now is that ffmpeg is using backspace characters or
similar to mess with stdout. Not sure what's happening here.


Answer: I suggest to use [sh](https://pypi.python.org/pypi/sh) module. It is very nice
piece of software which wraps subprocesses in python and gives you pythonic,
beautiful interface which you will love. Look at the
[docs](http://amoffat.github.com/sh/).

If you really do not want `sh` module, use
[communicate](http://docs.python.org/2/library/subprocess.html#subprocess.Popen.communicate)
method of `Popen`

