title: How to parse compressed sitemap using python without downloading it to disk?
body: <p>I want to parse compressed sitemap like www.example.com/sitemap.xml.gz and collect all the urls in sitemap without downloading sitemap.xml.gz.  </p>
  
  <p>there are ways to parse it after downloading sitemap.xml.gz and de-compressing it with help of <code>lxml</code> or <code>beautifulsoup</code> etc.  </p>
  
  <pre><code>def parse_sitemap_gz(url):
      r = requests.get(url, stream=True)
      if 200 != r.status_code:
      return False
      file_name = url.split('/')[-1]
  
      # download the sitemap file
      with open(file_name, 'wb') as f:
      if not r.ok:
          print 'error in %s'%(url)
      for block in r.iter_content(1024):
          if not block:
             break
          f.write(block) # can I parse it without writing to file
          f.flush()
  
      # decompress gz file
      subprocess.call(['gunzip', '-f', file_name])
  
      # parse xml file
      page = lxml.html.parse(file_name[0:-3])
      all_urls = page.xpath('//url/loc/text()')
      #print all_urls
  
      # delete sitemap file now
      subprocess.call(['rm', '-rf', file_name[0:-3]])
      return all_urls
  </code></pre>
  
  <p>in this code I am writing compressed sitemap to file. my intension is not to write anything to file.<br>
  for learning and creating intelligent version of above code, how can I parse it with concept of decompressing gzip streams so I wont need to download file or write it to file?</p>