title: get words from large file, using low memory in python
body: <p>I need to iterate over the words in a file. The file could be very big (over 1TB), the lines could be very long (maybe just one line). Words are English, so reasonable in size. So I don't want to load in the whole file or even a whole line.</p>
  
  <p>I have some code that works, but may explode if lines are to long (longer than ~3GB on my machine).</p>
  
  <pre><code>def words(file):
      for line in file:
          words=re.split("\W+", line)
          for w in words:
              word=w.lower()
              if word != '': yield word
  </code></pre>
  
  <p>Can you tell be how I can, simply, rewrite this iterator function so that it does not hold more than needed in memory?</p>