title: What is the fastest way to read 10 GB file from the disk?
body: <p>We need to read and count different types of messages/run
  some statistics on a 10 GB text file, e.g a <a href="http://en.wikipedia.org/wiki/Financial_Information_eXchange" rel="noreferrer">FIX</a> engine
  log. We use Linux, 32-bit, 4 CPUs, Intel, coding in Perl but
  the language doesn't really matter.</p>
  
  <p>I have found some interesting tips in Tim Bray's 
  <a href="http://www.tbray.org/ongoing/When/200x/2007/09/20/Wide-Finder" rel="noreferrer">WideFinder project</a>. However, we've found that using memory mapping
  is inherently limited by the 32 bit architecture.</p>
  
  <p>We tried using multiple processes, which seems to work
  faster if we process the file in parallel using 4 processes
  on 4 CPUs. Adding multi-threading slows it down, maybe
  because of the cost of context switching. We tried changing
  the size of thread pool, but that is still slower than
  simple multi-process version.</p>
  
  <p>The memory mapping part is not very stable, sometimes it
  takes 80 sec and sometimes 7 sec on a 2 GB file, maybe from
  page faults or something related to virtual memory usage.
  Anyway, Mmap cannot scale beyond 4 GB on a 32 bit
  architecture.</p>
  
  <p>We tried Perl's <a href="http://search.cpan.org/dist/IPC-Mmap" rel="noreferrer">IPC::Mmap</a> and <a href="http://search.cpan.org/dist/Sys::Mmap" rel="noreferrer">Sys::Mmap</a>. Looked
  into Map-Reduce as well, but the problem is really I/O
  bound, the processing itself is sufficiently fast.</p>
  
  <p>So we decided to try optimize the basic I/O by tuning
  buffering size, type, etc.</p>
  
  <p>Can anyone who is aware of an existing project where this
  problem was efficiently solved in any language/platform
  point to a useful link or suggest a direction?</p>