title: NoSQL for filesystem storage organization and replication?
body: <p>We've been discussing design of a data warehouse strategy within our group for meeting testing, reproducibility, and data syncing requirements.  One of the suggested ideas is to adapt a NoSQL approach using an <a href="http://nosql-database.org/" rel="nofollow noreferrer">existing tool</a> rather than try to re-implement a whole lot of the same on a file system.  I don't know if a NoSQL approach is even the best approach to what we're trying to accomplish but perhaps if I describe what we need/want you all can help.</p>
  
  <ol>
  <li>Most of our files are large, 50+ Gig in size, held in a proprietary, third-party format.  We need to be able to access each file by a name/date/source/time/artifact combination.  Essentially a key-value pair style look-up.</li>
  <li>When we query for a file, we don't want to have to load all of it into memory.  They're really too large and would swamp our server.  We want to be able to somehow get a reference to the file and then use a proprietary, third-party API to ingest portions of it.</li>
  <li>We want to easily add, remove, and export files from storage.</li>
  <li>We'd like to set up automatic file replication between two servers (we can write a script for this.)  That is, sync the contents of one server with another.  We don't need a distributed system where it only appears as if we have one server.  We'd like complete replication.  </li>
  <li>We also have other smaller files that have a tree type relationship with the Big files.  One file's content will point to the next and so on, and so on.  It's not a "spoked wheel," it's a full blown tree.</li>
  </ol>
  
  <p>We'd prefer a Python, C or C++ API to work with a system like this but most of us are experienced with a variety of languages.  We don't mind as long as it works, gets the job done, and saves us time.  What you think?  Is there something out there like this?</p>