What is the best way to capture output from a process using python?
Question: I am using python's `subprocess` module to start a new process. I would like
to capture the output of the new process in real time so I can do things with
it (display it, parse it, etc.). I have seen many examples of how this can be
done, some use custom file-like objects, some use `threading` and some attempt
to read the output until the process has completed.

[**File Like Objects Example (click
me)**](http://stackoverflow.com/questions/2297933/creating-a-custom-sys-
stdout-class)

  * I would prefer not to use custom file-like objects because I want to allow users to supply their own values for `stdin`, `stdout` and `stderr`.

[**Threading Example (click
me)**](http://codereview.stackexchange.com/questions/6567/how-to-redirect-a-
subprocesses-output-stdout-and-stderr-to-logging-module/17959#17959)

  * I do not really understand why threading is required so I am reluctant to follow this example. If someone can explain why the threading example makes sense I would be happy listen. However, this example also restricts users from supplying their own `stdout` and `stderr` values.

**Read Output Example (see below)**

The example which makes the most sense to me is to read the `stdout`, `stderr`
until the process has finished. Here is some example code:

    
    
    import subprocess
    
    # Start a process which prints the options to the python program.
    process = subprocess.Popen(
                               ["python", "-h"],
                               bufsize=1,
                               stdin=subprocess.PIPE,
                               stdout=subprocess.PIPE,
                               stderr=subprocess.PIPE,
                               )    
    
    # While the process is running, display the output to the user.
    while True:
    
        # Read standard output data.
        for stdout_line in iter(process.stdout.readline, ""):
    
            # Display standard output data.
            sys.stdout.write(stdout_line)
    
        # Read standard error data.
        for stderr_line in iter(process.stderr.readline, ""):
    
            # Display standard error data.
            sys.stderr.write(stderr_line)
    
        # If the process is complete - exit loop.
        if process.poll() != None:
            break
    

My question is,

**Q.** Is there a recommended approach for capturing the output of a process
using python?


Answer: First, your design is a bit silly, since you can do the same thing like this:

    
    
    process = subprocess.Popen(
                               ["python", "-h"],
                               bufsize=1,
                               stdout=sys.stdout,
                               stderr=sys.stderr
                               )
    

… or, even better:

    
    
    process = subprocess.Popen(
                               ["python", "-h"],
                               bufsize=1
                               )
    

However, I'll assume that's just a toy example, and you might want to do
something more useful.

* * *

The main problem with your design is that it won't read anything from `stderr`
until `stdout` is done.

Imagine you're driving an MP3 player that prints each track name to stdout,
and logging info to stderr, and you want to play 10 songs. Do you really want
to wait 30 minutes before displaying any of the logging to your users?

If that _is_ acceptable, then you might as well just use `communicate`, which
takes care of all of the headaches for you.

Plus, even if it's acceptable for your model, are you sure you can queue up
that much unread data in the pipe without it blocking the child? On every
platform?

Just breaking up the loop to alternate between the two won't help, because you
could end up blocking on `stdout.readline()` for 5 minutes while `stderr` is
piling up.

So that's why you need some way to read from both at once.

* * *

How do you read from two pipes at once?

This is the same problem (but smaller) as handling 1000 network clients at
once, and it has the same solutions: threading, or multiplexing (and the
various hybrids, like doing green threads on top of a multiplexor and event
loop, or using a threaded proactor, etc.).

The best sample code for the threaded version is
[`communicate`](http://hg.python.org/cpython/file/7f176a45211f/Lib/subprocess.py#l880)
from the 3.2+ source code. It's a little complicated, but if you want to
handle all of the edge cases properly on both Windows and Unix there's really
no avoiding a bit of complexity.

For multiplexing, you can use the
[`select`](http://docs.python.org/3/library/select.html) module, but keep in
mind that this only works on Unix (you can't `select` on pipes on Windows),
and it's buggy without 3.2+ (or the `subprocess32` backport), and to really
get all the edge cases right you need to add a signal handler to your
`select`. Unless you really, really don't want to use threading, this is the
harder answer.

But the _easy_ answer is to use someone else's implementation. There are a
dozen or more modules on PyPI specifically for async subprocesses.
Alternatively, if you already have a good reason to write your app around an
event loop, just about every modern event-loop-driven async networking library
(including the stdlib's
[`asyncio`](http://docs.python.org/dev/library/asyncio.html)) includes
subprocess support out of the box, that works on both Unix and Windows.

* * *

> Is there a recommended approach for capturing the output of a process using
> python?

It depends on who you're asking; a thousand Python developers might have a
thousand different answers… or at least half a dozen. If you're asking what
the core devs would recommend, I can take a guess:

If you don't need to capture it asynchronously, use `communicate` (but make
sure to upgrade to at least 3.2 for important bug fixes). If you do need to
capture it asynchronously, use `asyncio` (which requires 3.4).

