title: mmap and memory usage
body: <p>I am writing a program that receives huge amounts of data (in pieces of different sizes) from the network, processes them and writes them to memory. Since some pieces of data can be very large, my current approach is limiting the buffer size used. If a piece is larger than the maximum buffer size, I write the data to a temporary file and later read the file in chunks for processing and permanent storage.</p>
  
  <p>I'm wondering if this can be improved. I've been reading about mmap for a while but I'm not one hundred percent sure if it can help me. My idea is to use mmap for reading the temporary file. Does this help in any way? The main thing I'm concerned about is that an occasional large piece of data should not fill up my main memory causing everything else to be swapped out.</p>
  
  <p>Also, do you think the approach with temporary files is useful? Should I even be doing that or, perhaps, should I trust the linux memory manager to do the job for me? Or should I do something else altogether?</p>