title: Write multiple streams to a single file without knowing the length of the streams?
body: <p>For performance of reading and writing a large dataset, we have multiple threads compressing and writing out separate files to a SAN.  I'm making a new file spec that will instead have all these files appended together into a single file. I will refer to each of these smaller blocks of a data as a subset.</p>
  
  <p>Since each subset will be an unknown size after compression there is no way to know what byte offset to write to. Without compression each writer can write to a predictable address.</p>
  
  <p>Is there a way to append files together on the file-system level without requiring a file copy?  </p>
  
  <p>I'll write an example here of how I would expect the result to be on disk. Although I'm not sure how helpful it is to write it this way.</p>
  
  <pre><code>single-dataset.raw
  [header 512B][data1-45MB][data2-123MB][data3-4MB][data5-44MB] 
  </code></pre>
  
  <p>I expect the SAN to be NTFS for now in case there are any special features of certain file-systems.</p>
  
  <p>If I make the subsets small enough to fit into ram, I will know the size after compression, but keeping them smaller has other performance drawbacks.</p>