what would be a good approach for multithreading recursive traversal over directory tree?
<p>I'm thinking of a good approach for multithreaded recursive traversal over a directory tree.</p>

<p>What I'm doing right now is taking a the folders under a root directory and running a recursive traversal on each folder in a separate thread.</p>

<p>This improved the performance quite a bit, but I still have a "long tail" problem - large folders take a lot of time.</p>

<p>What approach would be better than this?</p>

<p>I'm using python and java for this.</p>

<p>thanks.</p>

<p>EDIT:
I don't need to load the files into memrory. just handle the file paths and maybe take some metadata info  like file size.</p>
