Speed up Crawling Every File C:\ and hashing using Java 8 I am wondering if anyone can help me I am trying to read every file in every directory and create a hash of the file. The code works fine but is extremely slow, I wondering what I can do increase the speed and the performance of the below

    public static void main(String[] args) throws IOException, NoSuchAlgorithmException {
    Test fw = new Test();
    fw.crawl("c:\\");
}

public void crawl(String path) throws IOException, NoSuchAlgorithmException {

    File root = new File(path);
    File[] list = root.listFiles();

    if (list == null) {
        return;
    }

    for (File f : list) {

        if (f.isDirectory()) {
            crawl(f.getAbsolutePath());
            System.out.println("Name:" + f.getName());
            System.out.println("Dir:" + f.getAbsoluteFile());
        } else {
            System.out.println("Name:" + f.getName());
            System.out.println("File:" + f.getAbsoluteFile());

            InputStream input = new BufferedInputStream(new FileInputStream(f));
            MessageDigest md5 = MessageDigest.getInstance("MD5");
            int tempByte;
            while ((tempByte = input.read()) != -1) {
                md5.update((byte) tempByte);
            }
            System.out.println("Hash:" + Arrays.toString(md5.digest()));
        }
    }
}
The main drawbacks of the solution you are trying to apply are:


The I/O streams are never closed: resources are leaking severely.
The InputStream reads data per byte and updates the MessageDigest with a single byte, but it would be more efficient when using a buffer.
The main logic body is using older File API, but there are newer tools since Java 7.
md5 instance is created for each file, but it can be reused by calling md5.reset().


Here what I can suggest to improve this solution.

The main call will be ridiculously short:

public void crawl2(String path) throws IOException {
  Files.walkFileTree(Paths.get(path), new HashingFileCrawler());
}


Files.walkFileTree needs an instance of FileVisitor and it executes all the boilerplate of traversing the file hierarchy, we need only to implement our logic. Let's call our implementor HashingFileCrawler:

public class HashingFileCrawler extends SimpleFileVisitor&lt;Path&gt; {

  private final MessageDigest md5; // only one instance

  public HashingFileCrawler() {
    try {
      this.md5 = MessageDigest.getInstance("MD5");
    } catch (NoSuchAlgorithmException ex) {
      // if we enter this catch block, there is definitely something wrong with the running JVM. "MD5" algo should exist by default.
      throw new IllegalStateException(ex);
    }
  }

  @Override
  public FileVisitResult preVisitDirectory(Path dir, BasicFileAttributes attrs) {
    System.out.println("Dir: " + dir.toString());
    return FileVisitResult.CONTINUE;
  }

  @Override
  public FileVisitResult visitFile(Path file, BasicFileAttributes attrs) throws IOException {
    System.out.println("File: " + file.toString());
    final byte[] buffer = new byte[2048];
    try (InputStream fis = Files.newInputStream(file);
         BufferedInputStream bis = new BufferedInputStream(fis)) {
      int nbRead = bis.read(buffer); // read file data into the buffer for exchange with md5
      while (nbRead &gt; -1) {
        md5.update(buffer, 0, nbRead);
        nbRead = bis.read(buffer);
      }
      System.out.println("Hash:" + Arrays.toString(md5.digest()));
      md5.reset();  // reset MD for reuse for next file
      return FileVisitResult.CONTINUE;
    }
  }
}


preVisitDirectory is called before crawling the files in a directory: it corresponds to the original code under condition if (f.isDirectory()).

visitFile wraps the main logic of producing a hash for a file, under else in the original code.

The size of the byte buffer (2048 here) can be adjusted: try to play with different values to see whether the perfs change.