title: python + urllib2: streaming ends prematurely
body: <p>I'm trying to process a large gzip file pulled from the internet in python using urllib2 and zlib and techniques from these two stackoverflow questions:</p>
  
  <ul>
  <li><a href="https://stackoverflow.com/questions/2423866/python-decompressing-gzip-chunk-by-chunk">Python decompressing gzip chunk-by-chunk</a></li>
  <li><a href="https://stackoverflow.com/questions/1517616/stream-large-binary-files-with-urllib2-to-file">Stream large binary files with urllib2 to file</a></li>
  </ul>
  
  <p>This works great, except that after each chunk of the file is read, I need to do some operations on the resultant string which involve a lot of splitting and iterating. This takes some time and when the code goes to do the next <code>req.read()</code>, it returns nothing, and the program ends, having only read the first chunk. </p>
  
  <p>If I comment out the other operations, the whole file is read and decompressed. Code:</p>
  
  <pre><code>d = zlib.decompressobj(16+zlib.MAX_WBITS)
  CHUNK = 16 * 1024
  url = 'http://foo.bar/foo.gz'
  req = urllib2.urlopen(url)
  while True:
      chunk = req.read(CHUNK)
      if not chunk:
          print "DONE"
          break
      s = d.decompress(chunk)
      # ...
      # lots of operations with s
      # which might take a while
      # but not more than 1-2 seconds
  </code></pre>
  
  <p>Any ideas? </p>
  
  <p><strong>Edit:</strong>
  This turned out to be a bug elsewhere in the program, NOT in the urllib2/zlib handling. Thanks to everyone who helped. I can recommend the pattern used in the code above if you need to handle large gzip files.</p>