title: pipe large amount of data to stdin while using subprocess.Popen
body: <p>I'm kind of struggling to understand what is the python way of solving this simple problem.</p>
  
  <p>My problem is quite simple. If you use the follwing code it will hang. This is well documented in the subprocess module doc.</p>
  
  <pre><code>import subprocess
  
  proc = subprocess.Popen(['cat','-'],
                          stdin=subprocess.PIPE,
                          stdout=subprocess.PIPE,
                          )
  for i in range(100000):
      proc.stdin.write('%d\n' % i)
  output = proc.communicate()[0]
  print output
  </code></pre>
  
  <p>Searching for a solution (there is a very insightful thread, but I've lost it now) I found this solution (among others) that uses an explicit fork:</p>
  
  <pre><code>import os
  import sys
  from subprocess import Popen, PIPE
  
  def produce(to_sed):
      for i in range(100000):
          to_sed.write("%d\n" % i)
          to_sed.flush()
      #this would happen implicitly, anyway, but is here for the example
      to_sed.close()
  
  def consume(from_sed):
      while 1:
          res = from_sed.readline()
          if not res:
              sys.exit(0)
              #sys.exit(proc.poll())
          print 'received: ', [res]
  
  def main():
      proc = Popen(['cat','-'],stdin=PIPE,stdout=PIPE)
      to_sed = proc.stdin
      from_sed = proc.stdout
  
      pid = os.fork()
      if pid == 0 :
          from_sed.close()
          produce(to_sed)
          return
      else :
          to_sed.close()
          consume(from_sed)
  
  if __name__ == '__main__':
      main()
  </code></pre>
  
  <p>While this solution is conceptually very easy to understand, it uses one more process and stuck as too low level compared to the subprocess module (that is there just to hide this kind of things...).</p>
  
  <p>I'm wondering: is there a simple and clean solution using the subprocess module that won't hung or to implement this patter I have to do a step back and implement an old-style select loop  or an explicit fork?</p>
  
  <p>Thanks</p>