title: How to write a live data set to disk with async I/O?
body: <p>I am new to developing in node.js (though relatively experienced at client-side javascript) and I'm running into lots of questions about good practices when dealing with asynchronous operations in node.js.</p>
  
  <p>My specific issue (though I imagine this is a fairly general purpose topic) is that I have a node.js app (running on a Raspberry Pi) that is recording the readings from several temperature probes every 10 seconds to an in memory data structure.  This works just fine.  The data accumulates over time in memory and, as it accumulates and reaches a particular size threshold, the data is regularly aged (keeping only the last N days of data) to keep it from growing beyond a certain size.  This temperature data is used to control some other appliances.</p>
  
  <p>Then, I have a separate interval timer that writes this data out to disk every so often (to persist it if the process crashes).  I'm using async node.js (<code>fs.open()</code>, <code>fs.write()</code> and <code>fs.close()</code>) disk IO to write the data out to disk.</p>
  
  <p>And, because of the async nature of the disk IO, it occurs to me that the very data structure I'm trying to write to disk may get modified right in the middle of me writing it out to disk.  That would potentially be a bad thing.  If data is only appended to the data structure while writing out to disk, that won't actually cause a problem with the way I'm writing the data, but there are some circumstances where earlier data can be modified as new data is being recorded and that would really mess with the integrity of what I'm in the middle of writing to disk.</p>
  
  <p>I can think of all sorts of somewhat ugly safeguards I could put in my code such as:</p>
  
  <ol>
  <li>Switch to synchronous IO to write the data to disk (don't really want to do that for server responsiveness reasons).</li>
  <li>Set a flag when I started writing data and don't record any new data while that flag is set (causes me to lose the recording of data during the write).</li>
  <li>More complicated versions of option 2 where I set the flag and when the flag is set, new data goes in a separate, temporary data structure that when the file IO is done is then merged with the real data (doable, but seems ugly).</li>
  <li>Take a snapshot copy of the original data and take your time to write that copy to disk knowing that nobody else will be modifying the copy.  I don't want to do this because the data set is relatively large and I'm in a limited memory environment (Raspberry PI).</li>
  </ol>
  
  <p>So, my question is what are design patterns for writing a large data set with async IO when other operations may want to modify that data during the async IO?  Are there more general purpose ways of handling my issue than the specific work-arounds listed above?</p>