Persistent python subprocess
Question: Is there a way to make a subprocess call in python "persistent"? I'm calling a
program that takes a while to load multiple times. So it would be great if I
could just leave that program open and communicate with it without killing it.

The cartoon version of my python script looks like this:

    
    
    for text in textcollection:
        myprocess = subprocess.Popen(["myexecutable"],
                    stdin = subprocess.PIPE, stdout = subprocess.PIPE,
                    stderr = None)
        myoutputtext, err = myprocess.communicate(input=text)
    

I need to process each text separately, so joining it all into one large text
file and processing it once is not an option.

Preferably, if there's an option like this

    
    
    myprocess = subprocess.Popen(["myexecutable"],
                stdin = subprocess.PIPE, stdout = subprocess.PIPE,
                stderr = None)    for text in textcollection:
    for text in textcollection:
        myoutputtext, err = myprocess.communicate(input=text)
    

where I can leave the process open, I'd really appreciate it.


Answer: You can use `myprocess.stdin.write()` and `myprocess.stdout.read()` to
communicate with your subprocess, you just need to be careful to make sure you
handle buffering correctly to prevent your calls from blocking.

If the output from your subprocess is well-defined, you should be able to
reliably communicate with it using line-buffering and
`myprocess.stdout.readline()`.

Here is an example:

    
    
    >>> p = subprocess.Popen(['cat'], bufsize=1, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
    >>> p.stdin.write('hello world\n')
    >>> p.stdout.readline()
    'hello world\n'
    >>> p.stdout.readline()        # THIS CALL WILL BLOCK
    

An alternative to this method for Unix is to put the file handle in non-
blocking mode, which will allow you to call functions like
`myprocess.stdout.read()` and have it return data if any is available, or
raise an `IOError` if there isn't any data:

    
    
    >>> p = subprocess.Popen(['cat'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
    >>> import fcntl, os
    >>> fcntl.fcntl(p.stdout.fileno(), fcntl.F_SETFL, os.O_NONBLOCK)
    0
    >>> p.stdout.read()         # raises an exception instead of blocking
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    IOError: [Errno 11] Resource temporarily unavailable
    

This would allow you to do something like this:

    
    
    fcntl.fcntl(p.stdout.fileno(), fcntl.F_SETFL, os.O_NONBLOCK)
    for text in textcollection:
        myprocess.stdin.write(text + '\n')
        while True:
            myoutputtext = ''
            try:
                myoutputtext += myprocess.stdout.read()
            except IOError:
                pass
            if validate_output(myoutputtext):
                break
            time.sleep(.1)    # short sleep before attempting another read
    

In this example, `validate_output()` is a function you would need to write
that returns `True` if the data you have received so far is all of output that
you expect to get.

