title: How to efficiently parse large text files in Ruby
body: <p>I'm writing an import script that processes a file that has potentially hundreds of thousands of lines (log file). Using a very simple approach (below) took enough time and memory that I felt like it would take out my MBP at any moment, so I killed the process.</p>
  
  <pre><code>#...
  File.open(file, 'r') do |f|
    f.each_line do |line|
      # do stuff here to line
    end
  end
  </code></pre>
  
  <p>This file in particular has 642,868 lines:</p>
  
  <pre><code>$ wc -l nginx.log                                                                                                                                        /code/src/myimport
    642868 ../nginx.log
  </code></pre>
  
  <p>Does anyone know of a more efficient (memory/cpu) way to process each line in this file?</p>
  
  <p><strong>UPDATE</strong></p>
  
  <p>The code inside of the <code>f.each_line</code> from above is simply matching a regex against the line. If the match fails, I add the line to a <code>@skipped</code> array. If it passes, I format the matches into a hash (keyed by the "fields" of the match) and append it to a <code>@results</code> array.</p>
  
  <pre><code># regex built in `def initialize` (not on each line iteration)
  @regex = /(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}) - (.{0})- \[([^\]]+?)\] "(GET|POST|PUT|DELETE) ([^\s]+?) (HTTP\/1\.1)" (\d+) (\d+) "-" "(.*)"/
  
  #... loop lines
  match = line.match(@regex)
  if match.nil?
    @skipped &lt;&lt; line
  else
    @results &lt;&lt; convert_to_hash(match)
  end
  </code></pre>
  
  <p>I'm completely open to this being an inefficient process. I could make the code inside of <code>convert_to_hash</code> use a precomputed lambda instead of figuring out the computation each time. I guess I just assumed it was the line iteration itself that was the problem, not the per-line code.</p>