title: Apache Spark on HDFS: read 10k-100k of small files at once
body: <p>I could have up to 100 thousands of small files (each 10-50 KB). They all are stored at HDFS with block size 128 MB. I have to read them at once with Apache Spark, as below:</p>
  
  <pre><code>// return a list of paths to small files
  List&lt;Sting&gt; paths = getAllPaths(); 
  // read up to 100000 small files at once into memory
  sparkSession
      .read()
      .parquet(paths)
      .as(Encoders.kryo(SmallFileWrapper.class))
      .coalesce(numPartitions);
  </code></pre>
  
  <h2>Problem</h2>
  
  <p><a href="https://stackoverflow.com/questions/43849821/hdfs-behavior-on-lots-of-small-files-and-128-mb-block-size">The number of small files is not a problem from the perspective of memory consumption</a>. The problem is the speed of reading of that amount of files. It takes 38 seconds to read 490  small files, and 266 seconds to read 3420 files. I suppose it would take a lot to read 100.000 files. </p>
  
  <h2>Questions</h2>
  
  <p><em>Will HAR or sequence files speed up Apache Spark batch read of 10k-100k of small files? Why?</em></p>
  
  <p><em>Will HAR or sequence files slow down persisting of that small files? Why?</em></p>
  
  <h2>P.S.</h2>
  
  <p>Batch read is the only operation required for that small files, I don't need to read them by id or anything else.</p>