title: Write CSV Data to hard disk progressively to reduce memory usage
body: <p>I've written a multithreaded web crawler that gathers data to be eventually written to a very large csv file. The only problem is, the program is using a tremendous amount of memory at the moment (had it at 15gb+ of memory, all of my computer's physical memory is being used). I'm not sure how python is managing the memory in the background but the program seems to be functional nonetheless, but I do get these pauses frequently which I think are to do with either python or the system doing some "memory magic" in the background. The only other thing it could be is CPU usage.</p>
  
  <p>Anyway, I think it would be helpful to write the data to the hard disk progressively so to reduce the memory usage. I'm not doing any additional operations on the data, so I really don't need it in memory.</p>
  
  <p>I was thinking I could do something like:</p>
  
  <pre><code># out_queue is queue.Queue() object containing data to be written
  def csv_writer(out_queue):
  
      with open(file, 'w+') as csvfile:
          writer = csv.writer(csvfile,delimiter=',')
          while True:
              task = out_queue.get()
              writer.writerow(task)
              task.task_done()
  </code></pre>
  
  <p>But I'm not not sure how the <code>open</code> function works, reading the documentation it said that a file object is returned, but I'm not sure what that really means. So I suppose the main questions here are:</p>
  
  <p>If I'm using a <code>with open</code> statement, is the open file held in memory? Reading the documentation didn't particularly help me.</p>
  
  <p>I also read something about opening a file with buffering, would that be helpful in this instance?</p>
  
  <p>If you have any other tips, for those of you out there who have dealt with these kind of issues, is there a better way to resolve it? Or should I just leave it to python to handle regardless?</p>