Python - simple reading lines from a pipe
Question: I'm trying to read lines from a pipe and process them, but I'm doing something
silly and I can't figure out what. The producer is going to keep producing
lines indefinitely, like this:

producer.py

    
    
    import time
    
    while True:
        print 'Data'
        time.sleep(1)
    

The consumer just needs to check for lines periodically:

consumer.py

    
    
    import sys, time
    while True:
        line = sys.stdin.readline()
        if line:
            print 'Got data:', line
        else:
            time.sleep(1)
    

When I run this in the Windows shell as `python producer.py | python consumer.py`, it just sleeps forever (never seems to get data?) It seems that maybe the problem is that the producer never terminates, since if I send a finite amount of data then it works fine.

How can I get the data to be received and show up for the consumer? In the
real application, the producer is a C++ program I have no control over.


Answer: Some old versions of Windows simulated pipes through files (so they were prone
to such problems), but that hasn't been a problem in 10+ years. Try adding a

    
    
      sys.stdout.flush()
    

to the producer after the `print`, and also try to make the producer's stdout
unbuffered (by using `python -u`).

Of course this doesn't help if you have no control over the producer -- if it
buffers too much of its output you're still going to wait a long time.

Unfortunately - while there are many approaches to solve that problem on Unix-
like operating systems, such as pyexpect,
[pexpect](http://sourceforge.net/projects/pexpect/),
[exscript](http://code.google.com/p/exscript/), and
[paramiko](http://www.lag.net/paramiko/), I doubt any of them works on
Windows; if that's indeed the case, I'd try [Cygwin](http://www.cygwin.com/),
which puts enough of a Linux-like veneer on Windows as to often enable the use
of Linux-like approaches on a Windows box.

