title: Bufferization in GzipFile
body: <p>Imagine the following simple script:</p>
  
  <pre><code>def reader():
      for line in open('logfile.log'):
          # do some stuff here like splitting the line or filtering etc.
          yield some_new_line
  
  def writer(stream):
      with gzip.GzipFile('some_output_file.gz', 'w') as fh:
          for _s in stream:
              fh.write(_s+'\n')
  
  stream = reader()
  writer(stream)
  </code></pre>
  
  <p>So pretty simple - read lines using generators and write some result into a gzip file.</p>
  
  <p>But how to speed it up? The HDD seems to be a bottleneck. I saw I can use buffer size for reads - using <em>open(file, mode, buffer)</em> syntax. But I'm not quite sure it will work in my case (with generators).</p>
  
  <p>Also I didn't find any bufferization parameter for the gzip.GzipFile call. From the code, it's based on some bufferized class, but I don't see any further docs on that.</p>
  
  <p>I have a (crazy?) idea to create an explicit cache and replace open methods with it - so it will read the file in bigger chunks, say, by 8MB, and then perform splitting it by lines. As for writes, I thought to create a list of lines to write, collect them (say, 5000 lines), and then dump into the file.</p>
  
  <p>Am I trying to re-invent the wheel? I'm not satisfied with the performance the script currently has, so I'm trying to speed it up as much as possible.</p>
  
  <p>UPD. I have around 4-5 different parallel workers running. They all perform reads and writes. So I guess the HDD is jumping from one sector to another, and this is the reason why I want to implement some bufferization to dump the data periodically in big chunks.</p>
  
  <p>Thanks!</p>