title: spark sql : GC overhead limit exceeded when reading parquet partitioned files
body: <p>I am trying read existing parquet files from hdfs using spark sql for my POC, but hitting the OOM error. </p>
  
  <p>I need to read all the portioned files, for a given partition date. The partition is as follows : date/file_dir_id</p>
  
  <ol>
  <li>There are 1200 sub folders under date folder </li>
  <li>There are in total 234769 .parquet files under all these folders(Not a huge number)</li>
  <li>Overall size of all the .parquet files is 10g</li>
  </ol>
  
  <p>Parquet folder structure</p>
  
  <ol>
  <li>Date 
  
  <ul>
  <li>File_dir_1
  
  <ul>
  <li>File_1.parquet</li>
  <li>File_2.parquet</li>
  </ul></li>
  <li>File_dir_2
  
  <ul>
  <li>File_3.parquet</li>
  <li>File_3.parquet</li>
  </ul></li>
  </ul></li>
  </ol>
  
  <p>When I try to read the files for a particular date, numbers as mentioned above
  sparkSession.read().schema(someSchema).parquet(hdfs_path_folder/date=2018-03-05/*); // I get the below mentioned error.</p>
  
  <p>Other details </p>
  
  <ol>
  <li>Running in yarn / cluster mode  </li>
  <li>Spark 2.3  </li>
  <li>4 Node cluster(32 core/128 gb)</li>
  <li>5 Executors / 4 cores each</li>
  </ol>
  
  <p><strong>It doesn't help if I increase the driver memory or executor memory. Any help on how to overcome this please ?</strong></p>
  
  <h2>Error Details</h2>
  
  <pre><code>java.lang.OutOfMemoryError: GC overhead limit exceeded
  at java.util.Arrays.copyOf(Unknown Source)
  at java.lang.AbstractStringBuilder.ensureCapacityInternal(Unknown Source)
  at java.lang.AbstractStringBuilder.append(Unknown Source)
  at java.lang.StringBuffer.append(Unknown Source)
  at java.net.URI.appendSchemeSpecificPart(Unknown Source)
  at java.net.URI.toString(Unknown Source)
  at java.net.URI.&lt;init&gt;(Unknown Source)
  at org.apache.hadoop.fs.Path.initialize(Path.java:203)
  at org.apache.hadoop.fs.Path.&lt;init&gt;(Path.java:172)
  at org.apache.spark.sql.execution.datasources.InMemoryFileIndex$$anonfun$org$apache$spark$sql$execution$datasources$InMemoryFileIndex$$bulkListLeafFiles$3$$anonfun$7.apply(InMemoryFileIndex.scala:235)
  at org.apache.spark.sql.execution.datasources.InMemoryFileIndex$$anonfun$org$apache$spark$sql$execution$datasources$InMemoryFileIndex$$bulkListLeafFiles$3$$anonfun$7.apply(InMemoryFileIndex.scala:228)
  at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
  at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
  at scala.collection.mutable.ArraySeq.foreach(ArraySeq.scala:74)
  at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
  at scala.collection.AbstractTraversable.map(Traversable.scala:104)
  at org.apache.spark.sql.execution.datasources.InMemoryFileIndex$$anonfun$org$apache$spark$sql$execution$datasources$InMemoryFileIndex$$bulkListLeafFiles$3.apply(InMemoryFileIndex.scala:228)
  at org.apache.spark.sql.execution.datasources.InMemoryFileIndex$$anonfun$org$apache$spark$sql$execution$datasources$InMemoryFileIndex$$bulkListLeafFiles$3.apply(InMemoryFileIndex.scala:227)
  at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
  at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
  at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
  at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
  at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
  at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:186)
  at org.apache.spark.sql.execution.datasources.InMemoryFileIndex$.org$apache$spark$sql$execution$datasources$InMemoryFileIndex$$bulkListLeafFiles(InMemoryFileIndex.scala:227)
  at org.apache.spark.sql.execution.datasources.InMemoryFileIndex$.org$apache$spark$sql$execution$datasources$InMemoryFileIndex$$listLeafFiles(InMemoryFileIndex.scala:273)
  at org.apache.spark.sql.execution.datasources.InMemoryFileIndex$$anonfun$org$apache$spark$sql$execution$datasources$InMemoryFileIndex$$bulkListLeafFiles$1.apply(InMemoryFileIndex.scala:172)
  at org.apache.spark.sql.execution.datasources.InMemoryFileIndex$$anonfun$org$apache$spark$sql$execution$datasources$InMemoryFileIndex$$bulkListLeafFiles$1.apply(InMemoryFileIndex.scala:171)
  at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
  at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
  at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
  at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
  </code></pre>