Handling big text files in Python
<p>Basics are that I need to process 4gig text files on a per line basis.</p>

<p>using .readline() or for line in f is great for memory but takes ages to IO.  Would like to use something like yield, but that (I think) will chop lines.</p>

<p><strong>POSSIBLE ANSWER:</strong></p>

<pre><code>file.readlines([sizehint])¶
</code></pre>

<blockquote>
<pre><code>Read until EOF using readline() and return a list containing the lines
</code></pre>
  
  <p>thus read. If the optional sizehint
  argument is present, instead of
  reading up to EOF, whole lines
  totalling approximately sizehint bytes
  (possibly after rounding up to an
  internal buffer size) are read.
  Objects implementing a file-like
  interface may choose to ignore
  sizehint if it cannot be implemented,
  or cannot be implemented efficiently.</p>
</blockquote>

<p>Didn't realize you could do this!</p>
