title: Buffering to the hard disk
body: <p>I am receiving a large quantity of data at a fixed rate. I need to do some processing on this data on a different thread, but this may run slower than the data is coming in, so I need to buffer the data. Due to the quantity of data coming in the available RAM would be quickly exhausted, so it needs to overflow onto the hard disk. What I could do with is something like a filesystem-backed pipe, so the writer could be blocked by the filesystem, but not by the reader running too slowly.</p>
  
  <p>Here's a rough set of requirements:</p>
  
  <ul>
  <li>Writing should not be blocked by the reader running too slowly.</li>
  <li>If data is read slow enough that the available RAM is exhausted it should overflow to the filesystem. It's ok for writes to the disk to block.</li>
  <li>Reading should block if no data is available unless the stream has been closed by the writer.</li>
  <li>If the reader is able to keep up with the data then it should never hit the hard disk as the RAM buffer would be sufficient (nice but not essential).</li>
  <li>Disk space should be recovered as the data is consumed (or soon after).</li>
  </ul>
  
  <p>Does such a mechanism exist in Windows?</p>