Efficient way to iterate over list of files
<p>I am searching for an efficient way to iterate over thousands of files in one or more directories.</p>

<p>The only way to iterate over files in a directory seems to be <code>File.list*()</code> functions. These functions effectively load the entire list of files in some sort of Collection and then let the user iterate over it. This seems to be impractical in terms of time/memory consumption. I tried looking at commons-io and other similar tools. but they all ultimately call <code>File.list*()</code> somewhere inside. JDK7's <code>walkFileTree()</code> came close, but I don't have control over when to pick the next element.</p>

<p>I have over 150,000 files in a directory and after many -Xms/-Xmm trial runs I got rid of memory overflow issues. But the time it takes to fill the array hasn't changed.</p>

<p>I wish to make some sort of an Iterable class that uses opendir()/closedir() like functions to lazily load file names as required. Is there a way to do this?</p>

<p><strong>Update:</strong></p>

<p>Java 7 NIO.2 supports file iteration via <a href="http://docs.oracle.com/javase/7/docs/api/java/nio/file/DirectoryStream.html" rel="nofollow">java.nio.file.DirectoryStream</a>. It is an <a href="http://docs.oracle.com/javase/7/docs/api/java/lang/Iterable.html" rel="nofollow">Iterable</a> class. As for JDK6 and below, the only option is <code>File.list*()</code> methods.</p>
