title: os.walk() caching/speeding up
body: <p>I have a prototype server[0] that's doing an <code>os.walk()</code>[1] for each query a client[0] makes.</p>
  
  <p>I'm currently looking into ways of:</p>
  
  <ul>
  <li>caching this data in memory, </li>
  <li>speeding up queries, and </li>
  <li>hopefully allowing for expansion into storing metadata and data persistence later on.</li>
  </ul>
  
  <p>I find SQL <a href="http://mikehillyer.com/articles/managing-hierarchical-data-in-mysql/" rel="nofollow noreferrer">complicated</a> for tree structures, so I thought I would get some advice before actually committing to SQLite</p>
  
  <p>Are there any cross-platform, embeddable or bundle-able non-SQL databases that might be able to handle this kind of data? </p>
  
  <ul>
  <li>I have a small (10k-100k files) list.</li>
  <li>I have an extremely small amount of connections (maybe 10-20). </li>
  <li>I want to be able to scale to handling metadata as well.</li>
  </ul>
  
  <p>[0] the server and client are actually the same piece of software, this is a P2P application, that's designed to share files over a local trusted network with out a main server, using <code>zeroconf</code> for discovery, and twisted for pretty much everything else</p>
  
  <p>[1] query time is currently 1.2s with <code>os.walk()</code> on 10,000 files</p>
  
  <p>Here is the related function in my Python code that does the walking:</p>
  
  <pre><code>def populate(self, string):
      for name, sharedir in self.sharedirs.items():
          for root, dirs, files, in os.walk(sharedir):
              for dir in dirs:
                  if fnmatch.fnmatch(dir, string):
                      yield os.path.join(name, *os.path.join(root, dir)[len(sharedir):].split("/"))
              for file in files:
                  if fnmatch.fnmatch(file, string): 
                      yield os.path.join(name, *os.path.join(root, ile)[len(sharedir):].split("/"))
  </code></pre>