title: Storing a deep directory tree in a database
body: <p>I am working on a desktop application that is much like WinDirStat or voidtools' Everything - it maps hard drives, i.e. creates a deeply nested dictionary out of the directory tree.</p>
  
  <p>The desktop application should then store the directory trees in some kind of database, so that a web application can be used to browse them from root, depth level by depth level.</p>
  
  <p>Assume both applications run locally on the same machine for the time being.</p>
  
  <p>The question that comes to mind is how the data should be structured and what database should be utilized, considering:
  1) RAM consumption should be reasonable
  2) The time it takes to for the directory to be ready for viewing in the web application should be minimal</p>
  
  <p>P.S -
  My initial approach was serializing each file system node to JSON separately and inserting each into Mongo, with object references linking them to their children. That way the web application could easily load the data based on user demand.
  However, I am worried that making so many (a million, by average) independent inserts to Mongo will take a lot of time; if I make bulk inserts that means I have to keep each bulk in memory.</p>
  
  <p>I also considered dumping the entire tree as one deeply nested JSON, but the data is too large to be a Mongo document. GridFS can be used to store it, but then I would have the load the entire tree in the web application even though the deep nodes may not be of interest.</p>