How can I use Python to pipe stdin/stdout to Perl script
Question: This Python code pipes data through Perl script fine.

    
    
    import subprocess
    kw = {}
    kw['executable'] = None
    kw['shell'] = True
    kw['stdin'] = None
    kw['stdout'] = subprocess.PIPE
    kw['stderr'] = subprocess.PIPE
    args = ' '.join(['/usr/bin/perl','-w','/path/script.perl','<','/path/mydata'])
    subproc = subprocess.Popen(args,**kw)
    for line in iter(subproc.stdout.readline, ''):
        print line.rstrip().decode('UTF-8')
    

However, it requires that I first to save my buffers to a disk file
(/path/mydata). It's cleaner to loop through the data in Python code and pass
line-by-line to the subprocess like this:

    
    
    import subprocess
    kw = {}
    kw['executable'] = '/usr/bin/perl'
    kw['shell'] = False
    kw['stderr'] = subprocess.PIPE
    kw['stdin'] = subprocess.PIPE
    kw['stdout'] = subprocess.PIPE
    args = ['-w','/path/script.perl',]
    subproc = subprocess.Popen(args,**kw)
    f = codecs.open('/path/mydata','r','UTF-8')
    for line in f:
        subproc.stdin.write('%s\n'%(line.strip().encode('UTF-8')))
        print line.strip()  ### code hangs after printing this ###
        for line in iter(subproc.stdout.readline, ''):
            print line.rstrip().decode('UTF-8')
    subproc.terminate()
    f.close()
    

The code hangs with the readline after sending the first line to the
subprocess. I have other executables that use this exact same code perfectly.

My data files can be quite large (1.5 GB) Is there way to accomplish piping
the data without saving to file? I don't want to re-write the perl script for
compatibility with other systems.


Answer: Your code is blocking at the line:

    
    
    for line in iter(subproc.stdout.readline, ''):
    

because the only way this iteration can terminate is when EOF (end-of-file) is
reached, which will happen when the subprocess terminates. You don't want to
wait till the process terminates, however, you only want to wait till its
finished processing the line that was sent to it.

Futhermore, you're encountering issues with buffering as Chris Morgan has
already pointed out. Another [question on
stackoverflow](http://stackoverflow.com/questions/375427/non-blocking-read-on-
a-subprocess-pipe-in-python) discusses how you can do non-blocking reads with
subprocess. I've hacked up a quick and dirty adaptation of the code from that
question to your problem:

    
    
    def enqueue_output(out, queue):
        for line in iter(out.readline, ''):
            queue.put(line)
        out.close()
    
    kw = {}
    kw['executable'] = '/usr/bin/perl'
    kw['shell'] = False
    kw['stderr'] = subprocess.PIPE
    kw['stdin'] = subprocess.PIPE
    kw['stdout'] = subprocess.PIPE
    args = ['-w','/path/script.perl',]
    subproc = subprocess.Popen(args, **kw)
    f = codecs.open('/path/mydata','r','UTF-8')
    q = Queue.Queue()
    t = threading.Thread(target = enqueue_output, args = (subproc.stdout, q))
    t.daemon = True
    t.start()
    for line in f:
        subproc.stdin.write('%s\n'%(line.strip().encode('UTF-8')))
        print "Sent:", line.strip()  ### code hangs after printing this ###
        try:
            line = q.get_nowait()
        except Queue.Empty:
            pass
        else:
            print "Received:", line.rstrip().decode('UTF-8')
    
    subproc.terminate()
    f.close()
    

It's quite likely that you'll need to make modifications to this code, but at
least it doesn't block.

