title: c++: how to optimize IO?
body: <p>I am working on a mathematical problem that has the advantage of being able to "pre-compute" about half of the problem, save this information to file, and then reuse it many times to compute various 'instances' of my problem.  The difficulty is that uploading all of this information in order to solve the actual problem is a major bottleneck.</p>
  
  <p>More specifically: 
  I can pre-compute a huge amount of information - tons of probabilities (<code>long double</code>), a ton of <code>std::map&lt;int,int&gt;</code>, and much more - and save all this stuff to disk (several Gb).</p>
  
  <p>The second half of my program accepts an input argument <em>D</em>.  For each <em>D</em>, I need to perform a great many computations that involve a combination of the pre-computed data (from file), and some other data that are specific to <em>D</em> (so that the problem is different for each <em>D</em>).</p>
  
  <p>Sometimes I will need to pick out certain pieces of pre-computed information from the files.  Other times, I will need to upload every piece of data from a (large) file.</p>
  
  <p>Are there any strategies for making the IO faster?</p>
  
  <p>I already have the program parallelized (MPI, via <code>boost::mpi</code>) for other reasons, but regardless, accessing files on the disk is making my compute time unbearable.</p>
  
  <p>Any strategies or optimizations?</p>
  
  <p>Currently I am doing everything with <code>cstdio</code>,  i.e. no <code>iostream</code>.  Will that make a big difference?</p>