title: random reading on very large files with fgets seems to bring Windows caching at it's limits
body: <p>I have written a C/C++-program for Windows 7 - 64bit that works on very large files. In the final step it reads lines from an input-file (10GB+) and writes them to an output file. The access to the input-file is random, the writing is sequential.
  EDIT: Main reason for this approach is to reduce RAM usage.</p>
  
  <p>What I basically do in the reading part is this: (Sorry, very shortened and maybe buggy)</p>
  
  <pre><code>void seekAndGetLine(char* line, size_t lineSize, off64_t pos, FILE* filePointer){
    fseeko64(filePointer, pos, ios_base::beg);
    fgets(line, lineSize, filePointer);
  }
  </code></pre>
  
  <p>Normally this code is fine, not to say fast, but under some very special conditions it gets very slow. The behaviour doesn't seem to be deterministic, since the performance drops occure on different machines at other parts of the file or even don't occure at all. It even goes so far, that the program totally stops reading, while there are no disc-operations.</p>
  
  <p>Another sympthom seems to be the used RAM. My process keeps it's RAM steady, but the RAM used by the System grows sometimes very large. After using some RAM-Tools I found out, that the Windows Mapped File grows into several GBs. This behaviour also seems to depend on the hardware, since it occure on different machines at different parts of the process.</p>
  
  <p>As far as I can tell, this problem doesn't exist on SSDs, so it definitely has something to do with the responsetime of the HDD.</p>
  
  <p>My guess is that the Windows Caching gets somehow "wierd". The program is fast as long as the cache does it's work. But when Caching goes wrong, the behaviour goes either into "stop reading" or "grow cache size" and sometimes even both. Since I'm no expert for the windows caching algorithms, I would be happy to hear an explanation. Also, is there any way to get Windows out of C/C++ to manipulate/stop/enforce the caching. </p>
  
  <p>Since I'm hunting this problem for a while now, I've already tried some tricks, that didn't work out:</p>
  
  <ul>
  <li>filePointer = fopen(fileName, "rbR"); //Just fills the cache till the RAM is full</li>
  <li>massive buffering of the read/write, to stop getting the two into each others way</li>
  </ul>
  
  <p>Thanks in advance</p>