title: gawk running out of memory when going through large files: can i optimize my code?
body: <p>I'm using gawk to go through a large textual corpus (about 3-4GB, a compilation of ebooks) in order to print out every association of 3 words that appears at least 3 times, in order to produce linguistic statistics. Here is the code:</p>
  
  <p>content of file.awk:</p>
  
  <pre><code>BEGIN { RS="[^[:alnum:]]+" } 
  
  { w1 = w2; w2 = w3; w3 = $0 } 
  
  NR &gt; 2 { count[tolower(w1 " " w2 " " w3)]++ } 
  
  END { 
       for (phrase in count) {
           if (count[phrase] &gt;= 3) { 
               print phrase, count[phrase] 
           } 
       } 
  } 
  </code></pre>
  
  <p>command: gawk -f file.awk mytxtfile > output </p>
  
  <p>It works fine with small files (a few hundreds MB) but I can't get it to work with files bigger than 1GB: gawk eats all my RAM (8GB) in less than a minute, then starts eating my swap and the whole system eventually freezes.</p>
  
  <p>Do you know how I could optimize the code, even if it eventually takes much longer?</p>
  
  <p>Thank you very much</p>