title: Multithreaded Files Reading
body: <p>I need to read / parse a large binary file (4 ~ 6 GB) that comes in fixed chunks of 8192 bytes. My current solution involves streaming the file chunks using the Single Producer Multiple Consumer (SPMC) pattern.</p>
  
  <p><strong>EDIT</strong></p>
  
  <p>File size = N * 8192 Bytes</p>
  
  <p>All I am required to do is to do something to each of these 8192 bytes. The file is only required to be read once top down. </p>
  
  <p>Having thought that this should be an embarrassingly parallel problem, I would like to have <strong><em>X</em></strong> threads to read at equal ranges of (File Size / <strong><em>X</em></strong>) sizes independently. The threads do not need to communicate with each other at all.</p>
  
  <p>I've tried spawning <strong><em>X</em></strong> threads to open the same file and seek to their respective sections to process, however, this solution seems to have a problem with the due to HDD mechanical seeks and apparently performs worse than the SPMC solution.</p>
  
  <p>Would there be any difference if this method is used on the SSD instead?</p>
  
  <p>Or would it be more straight forward to just memory map the whole file and use <code>#pragma omp parallel for</code> to process the chunks? I suppose I would need sufficient enough RAM to do this?</p>
  
  <p>What would you suggest?</p>