How to interact with python's subprocess as a continuous session
Question: I need to implement a filter in python, which pick out specific outputs from a
Linux command-line dictionary tool. I need to:

  1. Get a set of words from a file
  2. Lookup each word: 1) if word not contains, skip it; 2) else if it is a verb, save the definition.

To test the code, I wrote two python file：

    
    
    # name.py
    import sys
    while True:
        print 'name => Q: what is your name?'
        sys.stdout.flush()
        name = raw_input()
        if name == 'Exit':
            break
        print 'name => A: your name is ' + name
        sys.stdout.flush()
    
    # test.py
    import subprocess
    child = subprocess.Popen(r'python name.py', 
                stdin = subprocess.PIPE, 
                stdout = subprocess.PIPE,
                stderr = subprocess.STDOUT, 
                shell = True)
    commandlist = ['Luke\n', 'Mike\n', 'Jonathan\n', 'Exit\n']
    for command in commandlist:
        child.stdin.write(command)
        child.stdin.flush()
        output = child.stdout.readline()
        print 'From PIPE: ' + output
    while child.poll() is None:
        print 'NOT POLL: ' + child.stdout.readline()
    child.wait()
    

The output is

    
    
    From PIPE: name => Q: what is your name?
    
    From PIPE: name => A: your name is Luke
    
    From PIPE: name => Q: what is your name?
    
    From PIPE: name => A: your name is Mike
    # these lines need to start with "From PIPE" ... 
    NOT POLL: name => Q: what is your name?
    
    NOT POLL: name => A: your name is Jonathan
    
    NOT POLL: name => Q: what is your name?
    
    NOT POLL: 
    

The later output is read during the `while` loop rather than the `for` loop in
`test.py`. What is the reason?

Due to the demand, I need to get the whole output each time it input a new
command. It seems like a dialog session. So the `subprocess.communicate()` is
useless there, for it always terminates current subprocess. How to implement
this demand?


Answer: The basic reason `subprocess` insists you use `.communicate()` is because it's
possible for deadlock to occur otherwise. Suppose you're writing to the
process's stdin, while the process is writing to its stdout. If the pipe
buffers fill up, the writes will block until a read occurs. Then you're both
waiting for each other and no progress can be made. There are several ways to
deal with this:

  1. Use separate threads. Assign one to stdin and the other to stdout. That way, if one pipe blocks, you're still servicing the other.
  2. Use `select` to multiplex over the pipes. Only interact with pipes which are ready for you. You should also enable `O_NONBLOCK` on the pipes using `fcntl`, so you don't accidentally fill the buffers. Used correctly, this will prevent the pipes from ever blocking, so you can't deadlock. **This doesn't work under Windows** , because you can only do `select` on sockets there.

