title: Traversing many files in Java with minimal memory usage
body: <p>I need to traverse a directory hierarchy containing about 20 million files in Java. Currently I'm using <a href="http://commons.apache.org/io/api-release/org/apache/commons/io/FileUtils.html#iterateFiles" rel="nofollow noreferrer"><code>FileUtils.iterateFiles</code></a> from Apache Commons-IO. This seems to work by loading the whole list into memory, which is slow (delaying the application startup time) and a <strong>huge</strong> memory hog (around 8GB). I was previously using my own recursive file iterator which had the same problem. </p>
  
  <p>I only need to process one file at a time (or, down the track, a handful from the front of the list in parallel), so it seems a little unnecessary to waste all this time and memory loading a complete list into memory. </p>
  
  <p>Java's <code>Iterator</code> class allows for minimal-memory footprint iterators of the kind that I need, but since the native features of the <code>java.io.File</code> class only provide eagerly-initialized arrays, it seems to be bizarrely difficult to take advantage of these.</p>
  
  <p>Does anyone have any suggestions for how I can traverse the file hierarchy without loading it all into memory in advance? </p>
  
  <p>Thanks to <a href="https://stackoverflow.com/a/10691278/1711188">this answer</a> I'm  now aware of the new Java 7 file API, which I think would solve my problem, but Java 7 is not really an option for me at this stage.</p>