title: Spark task hanging at [GC (Allocation Failure) ]
body: <p><strong>EDIT:</strong> Note: An executor will normally emit the message <code>[GC (Allocation Failure) ]</code> . It runs it because it is trying to allocate memory to the Executor, but the executor is full, so it will GC in attempt to make space when loading something new to the Executor. If your Executor does this in a loop, it may mean what you are trying to load into that Executor is too big.</p>
  
  <p>I am running Spark 2.2, Scala 2.11 on AWS EMR 5.8.0</p>
  
  <p>I'm trying to run a <code>count</code> operation on a Dataset that refuses to finish. What's frustrating, is that it is only hanging on one particular file. I run this job on a different file from S3, no problem - it finishes completely. The original CSV file is @18GB itself, and we run a transformation on it to turn the original CSV into a struct column, giving it one extra column.</p>
  
  <p>My environment's core slaves are 8 instance where each is:</p>
  
  <pre><code>r3.2xlarge
  16 vCore, 61 GiB memory, 160 SSD GB storage
  </code></pre>
  
  <p>My Spark session settings are:</p>
  
  <pre><code>implicit val spark = SparkSession
        .builder()
        .appName("MyApp")
        .master("yarn")
        .config("spark.speculation","false")
        .config("hive.metastore.uris", s"thrift://$hadoopIP:9083")
        .config("hive.exec.dynamic.partition", "true")
        .config("hive.exec.dynamic.partition.mode", "nonstrict")
        .config("mapreduce.fileoutputcommitter.algorithm.version", "2")
        .config("spark.dynamicAllocation.enabled", false)
        .config("spark.executor.cores", 5)
        .config("spark.executors.memory", "18G")
        .config("spark.yarn.executor.memoryOverhead", "2G")
        .config("spark.driver.memory", "18G")
        .config("spark.executor.instances", 23)
        .config("spark.default.parallelism", 230)
        .config("spark.sql.shuffle.partitions", 230)
        .enableHiveSupport()
        .getOrCreate()
  </code></pre>
  
  <p>The data comes from a CSV file: </p>
  
  <pre><code>val ds = spark.read
            .option("header", "true")
            .option("delimiter", ",")
            .schema(/* 2 cols: [ValidatedNel, and a stuct schema */)
            .csv(sourceFromS3)
            .as(MyCaseClass)
  
  val mappedDs:Dataset[ValidatedNel, MyCaseClass] = ds.map(...)
  
  mappedDs.repartition(230)
  
  val count = mappedDs.count() // never finishes
  </code></pre>
  
  <p>As expected, it spins up 230 tasks, and 229 finish, except one somewhere in the middle. See below - the first task just hangs forever, the middle one finishes no problem (though is odd - the size records/ratio is very different) - and the other 229 tasks look exactly the same as the finished one.</p>
  
  <pre><code>Index| ID |Attempt |Status|Locality Level|Executor ID / Host|                       Launch Time          |   Duration   |GC Time|Input Size / Records|Write Time | Shuffle Write Size / Records| Errors
  110   117   0   RUNNING     RACK_LOCAL     11 / ip-XXX-XX-X-XX.uswest-2.compute.internal 2019/10/01 20:34:01    1.1 h   43 min     66.2 MB / 2289538                0.0 B / 0   
  0     7     0   SUCCESS     PROCESS_LOCAL  9 / ip-XXX-XX-X-XXX.us-west-2.compute.internal 2019/10/01 20:32:10   1.0 s   16 ms      81.2 MB /293        5 ms         59.0 B / 1   &lt;-- this task is odd, but finishes
  1     8     0   SUCCESS     RACK_LOCAL      9 / ip-XXX-XX-X-XXX.us-west-2.compute.internal 2019/10/01 20:32:10  2.1 min     16 ms      81.2 MB /2894845        9 s          59.0 B / 1   &lt;- the other tasks are all similar to this one
  </code></pre>
  
  <p>Checking the stdout of the hanging tasks, I repeatedly see the following the never ends: </p>
  
  <pre><code>2019-10-01T21:51:16.055+0000: [GC (Allocation Failure) 2019-10-01T21:51:16.055+0000: [ParNew: 10904K-&gt;0K(613440K), 0.0129982 secs]2019-10-01T21:51:16.068+0000: [CMS2019-10-01T21:51:16.099+0000: [CMS-concurrent-mark: 0.031/0.044 secs] [Times: user=0.17 sys=0.00, real=0.04 secs] 
   (concurrent mode failure): 4112635K-&gt;2940648K(4900940K), 0.4986233 secs] 4123539K-&gt;2940648K(5514380K), [Metaspace: 60372K-&gt;60372K(1103872K)], 0.5121869 secs] [Times: user=0.64 sys=0.00, real=0.51 secs] 
  </code></pre>
  
  <p>Another note is that before I call the count, I'm calling <code>repartition(230)</code> just priot to calling <code>count</code> on the <code>Dataset[T]</code> to insure equal distribution of data</p>
  
  <p>What is going on here?</p>