Python subprocesses don't output properly?
Question: I don't think I'm understanding python subprocess properly at all but here's a
simple example to illustrate a point I'm confused about:

    
    
    #!/usr/bin/env python
    import subprocess
    lookup_server = subprocess.Popen("nc -l 5050", shell=True)
    lookup_client = subprocess.Popen("nc localhost 5050", shell=True, stdin=subprocess.PIPE) 
    print lookup_client.poll()
    lookup_client.stdin.write("magic\n")
    print lookup_client.poll()                                                                        
    lookup_client.send_signal(subprocess.signal.SIGINT)
    print lookup_client.poll()
    lookup_server.wait()
    print "Lookup server terminated properly"
    

The output comes back as

    
    
    None
    None
    None
    

and never completes. Why is this? Also, if I change the first argument of
Popen to an array of all of those arguments, none of the nc calls execute
properly and the script runs through without ever waiting. Why does that
happen?

Ultimately, I'm running into a problem in a much larger program that does
something similar using netcat and another program running locally instead of
two versions of nc. Either way, I haven't been able to write to or read from
them properly. However, when I run them in the python console everything runs
as I would expect. All this has me very frustrated. Let me know if you have
any insights!

EDIT: I'm running this on Ubuntu Linux 12.04, when I `man nc`, I get the BSD
General Commands manual so I'm assuming this is BSD netcat.


Answer: The problem here is that you're sending `SIGINT` to the process. If you just
`close` the `stdin`, `nc` will close its socket and quit, which is what you
want.

It sounds like you're actually using `nc` for the client (although not the
server) in your real program, which means you have two easy fixes:

Instead of `lookup_client.send_signal(subprocess.signal.SIGINT)`, just do
`lookup_client.stdin.close()`. `nc` will see this as an EOF on its input, and
exit normally, at which point your server will also exit.

    
    
    #!/usr/bin/env python
    import subprocess
    lookup_server = subprocess.Popen("nc -l 5050", shell=True)
    lookup_client = subprocess.Popen("nc localhost 5050", shell=True, stdin=subprocess.PIPE) 
    print lookup_client.poll()
    lookup_client.stdin.write("magic\n")
    lookup_client.stdin.close()
    print lookup_client.poll()
    lookup_server.wait()
    print "Lookup server terminated properly"
    

When I run this, the most common output is:

    
    
    None
    None
    magic
    Lookup server terminated properly
    

Occasionally the second `None` is a `0` instead, and/or it comes after `magic`
instead of before, but otherwise, it's always all four lines. (I'm running on
OS X.)

For this simple case (although maybe not your real case), just use
[`communicate`](http://docs.python.org/2/library/subprocess.html#subprocess.Popen.communicate)
instead of trying to do it manually.

    
    
    #!/usr/bin/env python
    import subprocess
    lookup_server = subprocess.Popen("nc -l 5050", shell=True)
    lookup_client = subprocess.Popen("nc localhost 5050", shell=True, stdin=subprocess.PIPE) 
    print lookup_client.communicate("magic\n")
    lookup_server.wait()
    print "Lookup server terminated properly"
    

Meanwhile:

> Also, if I change the first argument of Popen to an array of all of those
> arguments, none of the nc calls execute properly and the script runs through
> without ever waiting. Why does that happen?

As [the
docs](http://docs.python.org/2/library/subprocess.html#subprocess.Popen) say:

> On Unix with `shell=True`… If args is a sequence, the first item specifies
> the command string, and any additional items will be treated as additional
> arguments to the shell itself.

So, `subprocess.Popen(["nc", "-l", "5050"], shell=True)` does `/bin/sh -c 'nc'
-l 5050`, and `sh` doesn't know what to do with those arguments.

You probably _do_ want to use an array of args, but then you have to get rid
of `shell=True`—which is a good idea anyway, because the shell isn't helping
you here.

One more thing:

    
    
    lookup_client.send_signal(subprocess.signal.SIGINT)
    print lookup_client.poll()
    

This may print either -2 or None, depending on whether the client has finished
responding to the `SIGINT` and been killed before you `poll` it. If you want
to actually get that -2, you have to call `wait` rather than `poll` (or do
something else, like loop until `poll` returns non-None).

Finally, why didn't your original code work? Well, sending `SIGINT` is
asynchronous; there's no guarantee as to when it might take effect. For one
example of what could go wrong, it could take effect before the client even
opens the socket, in which case the server is still sitting around waiting for
a client that never shows up.

You can throw in a `time.sleep(5)` before the `signal` call to test this—but
obviously that's not a real fix, or even an acceptable hack; it's only useful
for testing the problem. What you need to do is not kill the client until it's
done everything you want it to do. For complex cases, you'll need to build
some mechanism to do that (e.g., reading its stdout), while for simple cases,
`communicate` is already everything you need (and there's no reason to kill
the child in the first place).

