Java: list all files(10-20,000+) from single directory
<p>I want to list large number of files(10, 20 thousand or so) contained in a single directory, quickly and efficiently.
I have read quite a few posts especially over here explaining the short coming of Java to achieve such, basically due to the underlying filesystem (and that probably Java 7 has some answer to it).
Some of the posts here have proposed alternatives like native calls or piping etc and I do understand the best possible option under normal circumstances is the java call
- String[] sList = file.list(); which is only slightly better than file.listFiles();
Also, there was a suggestion for the use of multithreading(also Executor service).</p>

<p>Well, here the issue is I have very little practical know-how of how to code multithreading way. So my logic is bound to be incorrect. Still, I tried this way:</p>

<ol>
<li>created a list of few thread objects</li>
<li>Ran a loop of this list, called the .start() and immediately .sleep(500)</li>
<li>In the thread class, over-rode the run methos to include the .list()</li>
</ol>

<p>Something like this, Caller class - </p>

<pre><code>String[] strList = null;  
for (int i = 0; i &lt; 5; i++){  
   ThreadLister tL = new ThreadLister(fit);  
   threadList.add(tL);  
}  
for (int j = 0; j &lt; threadList.size(); j++) {  
   thread = threadList.get(j);  
   thread.start();  
   thread.sleep(500);  
}  
strList = thread.fileList;  
</code></pre>

<p>and the Thread class as -   </p>

<pre><code>public String[] fileList;  
public ThreadLister(File f) {  
   this.f = f;  
}  
public void run() {  
   fileList = f.list();  
}  
</code></pre>

<p>I might be way off here with multithreading, I guess that.
I would very much appreciate a solution to my requirement the multithreading. Added benefit is I would learn a bit more about practical multithreading.</p>

<blockquote>
  <p>Query Update</p>
</blockquote>

<p>Well, Obviously multithreading isn't going to help me(well I now realise its not actually a solution). Thank you for helping me to rule out threading.<br>
So I tried,<br>
1. <code>FileUtils.listFiles()</code> from apache commons - not much difference.<br>
2. Native call viz. <code>exec("cmd /c dir /B .\\Test")</code> - here this executes fast but then when I read the Stream using a while loop that takes ages.  </p>

<p>What actually I require is filename depending upon a certain filter amongst about 100k files in single directory. So I am using like <code>File.list(new FileNameFilter())</code>.<br>
I believe FileNameFilter has no benefit, as it will try to match accordingly with all the files first and then give out the output.<br>
Yes, I understand, I need a different approach of storing these files. One option I can try is storing these files in multiple directories, I am yet to try this(I dont know if this will help enough) - <em>As suggested by Boris earlier</em>.<br>
What else can be a better option, will a native call on Unix ls with filename match work effectively. I know on windows it doesnt work, I mean unless we are searching in same directory  </p>

<p>Kind Regards</p>
