title: Extract contents of a large zip file
body: <p>I am trying to extract contents of a zip file of size ~500MB containing around 250K files.</p>
  
  <p>Here's what I am trying to do -</p>
  
  <pre><code>import java.io.File;
  import java.io.IOException;
  import java.util.ArrayList;
  import java.util.List;
  import java.util.concurrent.ExecutorService;
  import java.util.concurrent.Executors;
  
  import org.apache.commons.io.FileUtils;
  import org.apache.commons.io.IOUtils;
  
  import de.schlichtherle.truezip.file.TFile;
  import de.schlichtherle.truezip.file.TFileInputStream;
  
  public class ArchiveReaderExecutor {
  
      private final ExecutorService pool;
  
      public ArchiveReaderExecutor() {
          pool = Executors.newFixedThreadPool(8);
      }
  
      /**
       * Splits the archive file into list of lists as provided in the batch size
       * variable
       * 
       * @param archive
       * 
       * @return 
       */
      public List&lt;List&lt;TFile&gt;&gt; splitArchiveFile(final File archive) {
          final TFile tFile = new TFile(archive.getAbsolutePath());
          final ArrayList&lt;TFile&gt; individualFiles = new ArrayList&lt;TFile&gt;();
          recursivelyReadLeafnodes(tFile, individualFiles);
          final List&lt;List&lt;TFile&gt;&gt; returnList = new ArrayList&lt;List&lt;TFile&gt;&gt;();
  
          /*
           * Splitting the entire list into list of objects for batch processing
           */
          int count = 0;
          List&lt;TFile&gt; innerList = null;
  
          for (TFile splitFile : individualFiles) {
              if (count == 0) {
                  innerList = new ArrayList&lt;TFile&gt;();
                  returnList.add(innerList);
              }
  
              if (count &lt; 100) {
                  ++count;
              } else {
                  count = 0;
              }
              innerList.add(splitFile);
          }
          return returnList;
      }
  
      public List&lt;TFile&gt; recursivelyReadLeafnodes(TFile inputTFile,
              ArrayList&lt;TFile&gt; individualFiles) {
          TFile[] tfiles = null;
  
          if (inputTFile.isArchive() || inputTFile.isDirectory()) {
              tfiles = inputTFile.listFiles();
          } else {
              tfiles = new TFile[0];
              tfiles[0] = inputTFile;
          }
  
          for (final TFile tFile : tfiles) {
              if (tFile.isFile() &amp;&amp; !tFile.getName().startsWith(".")) {
                  individualFiles.add(tFile);
              } else if (tFile.isDirectory()) {
                  recursivelyReadLeafnodes(tFile, individualFiles);
              }
          }
  
          return individualFiles;
      }
  
      public void runExtraction() {
  
          File src = new File("Really_Big_File.zip");
          List&lt;List&lt;TFile&gt;&gt; files = splitArchiveFile(src);
          for (List&lt;TFile&gt; list : files) {
              pool.execute(new FileExtractorSavor(list));
          }
          pool.shutdown();
  
      }
  
  
      class FileExtractorSavor implements Runnable{
          List&lt;TFile&gt; files;
          public FileExtractorSavor(List&lt;TFile&gt; files) {
              this.files = files;
          }
          @Override
          public void run() {
              File file = null;
              TFileInputStream in = null;
              for (TFile tFile : files) {
                  try {
                      in = new TFileInputStream(tFile);
                      file = new File("Target_Location"+tFile.getName());
                      FileUtils.writeStringToFile(file, IOUtils.toString(in));
                  } catch (IOException e) {
                      e.printStackTrace();
                  } finally {
                      IOUtils.closeQuietly(in);
                  }
              }
  
          }
  
      }
  
      public static void main(String[] args) {
          new ArchiveReaderExecutor().runExtraction();
      }
  }
  </code></pre>
  
  <p>When I am running this code concurrently, there are a lot of threads in wait/blocked state, here's the thread dump:</p>
  
  <pre class="lang-none prettyprint-override"><code>"pool-1-thread-7" prio=5 tid=7fd8093dd000 nid=0x11d3f3000 waiting for monitor entry [11d3f2000]
     java.lang.Thread.State: BLOCKED (on object monitor)
      at de.schlichtherle.truezip.socket.ConcurrentInputShop$SynchronizedConcurrentInputStream.close(ConcurrentInputShop.java:223)
      - waiting to lock &lt;785460200&gt; (a de.schlichtherle.truezip.fs.archive.FsDefaultArchiveController$Input)
      at de.schlichtherle.truezip.io.DecoratingInputStream.close(DecoratingInputStream.java:79)
      at org.apache.commons.io.IOUtils.closeQuietly(IOUtils.java:178)
      at ArchiveReaderExecutor$FileExtractorSavor.run(ArchiveReaderExecutor.java:136)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
      at java.lang.Thread.run(Thread.java:695)
  
     Locked ownable synchronizers:
      - &lt;79ed370e0&gt; (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
  "pool-1-thread-5" prio=5 tid=7fd8093db800 nid=0x11d1ed000 waiting for monitor entry [11d1ec000]
     java.lang.Thread.State: BLOCKED (on object monitor)
      at de.schlichtherle.truezip.socket.ConcurrentInputShop$SynchronizedConcurrentInputStream.close(ConcurrentInputShop.java:223)
      - waiting to lock &lt;785460200&gt; (a de.schlichtherle.truezip.fs.archive.FsDefaultArchiveController$Input)
      at de.schlichtherle.truezip.io.DecoratingInputStream.close(DecoratingInputStream.java:79)
      at org.apache.commons.io.IOUtils.closeQuietly(IOUtils.java:178)
      at ArchiveReaderExecutor$FileExtractorSavor.run(ArchiveReaderExecutor.java:136)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
      at java.lang.Thread.run(Thread.java:695)
  
     Locked ownable synchronizers:
      - &lt;79ed46468&gt; (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
  </code></pre>
  
  <p>I also tried using:</p>
  
  <pre><code>TFile.cp_r(src, dst, TArchiveDetector.NULL, TArchiveDetector.NULL);
  </code></pre>
  
  <p>It took much longer as it was running on a single thread.</p>
  
  <p>My question, what's the fast, best and optimal way to extract contents of a zip file in java using TrueZip?</p>