title: I/O bound performance - speedup?
body: <p>I'm writing a program that involves analyzing CSV files of minimum 0.5GB (and maximum of over 20GB), I read from the CSV as follows with <code>fstream</code>, <code>while (getline(fin,line)) {}</code>, and doing an average of 17millisecs work on each comma separated record. Simple stuff. </p>
  
  <p>But, there are a LOT of records. So obviously, the program is I/O bound, but <strong>I was wondering whether I could improve the I/O performance</strong>. I can't resort to OpenMP as I would deal with CPU constraints, and buffering a file this large won't work either. So I might need some kind of pipeline...</p>
  
  <p>I have VERY little experience in multithreading in C++ and have never used dataflow frameworks. Could anyone point me in the right direction? </p>
  
  <hr>
  
  <p><strong>Update (12/23/14) :</strong> </p>
  
  <p>Thanks for all your comments. You are right, 17ms was a bit much... After doing a LOT of profiling (oh, the pain), I isolated the bottleneck as an iteration over a substring in each record (75 chars). I experimented with <code>#pragmas</code> but it simply isn't enough work to parallelize. the overhead of the function call was the main gripe - now 5.41μs per record, having shifted a big block. It's ugly, but faster.</p>
  
  <p>Thanks @ChrisWard1000 for your suggestions. Unfortunately I do not much have control over the hardware I'm using at the moment, but will profile with larger data sets (>20GB CSV) and see how I could introduce mmap/multithreaded parsing etc.</p>