title: Cleaner way to read/gunzip a huge file in python
body: <p>So I have some fairly gigantic .gz files - we're talking 10 to 20 gb each when decompressed.</p>
  
  <p>I need to loop through each line of them, so I'm using the standard:</p>
  
  <pre><code>import gzip
  f = gzip.open(path+myFile, 'r')
  for line in f.readlines():
      #(yadda yadda)
  f.close()
  </code></pre>
  
  <p>However, both the <code>open()</code> and <code>close()</code> commands take AGES, using up 98% of the memory+CPU. So much so that the program exits and prints <code>Killed</code> to the terminal. Maybe it is loading the entire extracted file into memory?</p>
  
  <p>I'm now using something like:</p>
  
  <pre><code>from subprocess import call
  f = open(path+'myfile.txt', 'w')
  call(['gunzip', '-c', path+myfile], stdout=f)
  #do some looping through the file
  f.close()
  #then delete extracted file
  </code></pre>
  
  <p>This works. But is there a cleaner way?</p>