title: Out of memory exception or worked node lost during the spark scala job
body: <p>I am executing a spark-scala job using spark-shell and the problem I am facing is, at the end of the final stage and final mapper like in stage 5 it allocates 50 and completed 49 very quickly and at the 50th it takes 5 minutes and says that out of memory and fails. I am using <code>SPARK_MAJOR_VERSION=2</code></p>
  
  <p>I am using the below command
  <code>spark-shell --master yarn --conf spark.driver.memory=30G --conf spark.executor.memory=40G --conf spark.shuffle.service.enabled=true --conf spark.dynamicAllocation.enabled=false --conf spark.sql.broadcastTimeout=36000 --conf spark.shuffle.compress=true --conf spark.executor.heartbeatInterval=3600s --conf spark.executor.instance=160</code></p>
  
  <p>In the above configuration I have tried the dynamic allocation to true and started the driver and executor memory from 1GB. I have the overall ram of 6.78TB and 1300 VCores(This is my entire hadoop hardware).</p>
  
  <p>The table I am reading is <code>40GB</code> and I am joining 6 tables to that 40GB table, so, overall might be 60GB. so spark is initializing 4 stages for this and in the final stage at the end it is failing. I am using the spark sql to execute SQL.</p>
  
  <p>Below are the errors:</p>
  
  <pre><code>19/04/26 14:29:02 WARN HeartbeatReceiver: Removing executor 2 with no recent heartbeats: 125967 ms exceeds timeout 120000 ms
  19/04/26 14:29:02 ERROR YarnScheduler: Lost executor 2 on worker03.some.com: Executor heartbeat timed out after 125967 ms
  19/04/26 14:29:02 WARN TaskSetManager: Lost task 5.0 in stage 2.0 (TID 119, worker03.some.com, executor 2): ExecutorLostFailure (executor 2 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 125967 ms
  19/04/26 14:29:02 WARN HeartbeatReceiver: Removing executor 1 with no recent heartbeats: 126225 ms exceeds timeout 120000 ms
  19/04/26 14:29:02 ERROR YarnScheduler: Lost executor 1 on ncednhpwrka0008.devhadoop.charter.com: Executor heartbeat timed out after 126225 ms
  19/04/26 14:29:02 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container marked as failed: container_e1223_1556277056929_0976_01_000003 on host: worker03.some.com. Exit status: 52. Diagnostics: Exception from container-launch.
  Container id: container_e1223_1556277056929_0976_01_000003
  Exit code: 52
  Shell output: main : command provided 1
  main : run as user is svc-bd-xdladmrw-dev
  main : requested yarn user is svc-bd-xdladmrw-dev
  Getting exit code file...
  Creating script paths...
  Writing pid file...
  Writing to tmp file /data/00/yarn/local/nmPrivate/application_1556277056929_0976/container_e1223_1556277056929_0976_01_000003/container_e1223_1556277056929_0976_01_000003.pid.tmp
  Writing to cgroup task files...
  Creating local dirs...
  Launching container...
  Getting exit code file...
  Creating script paths...
  
  
  Container exited with a non-zero exit code 52. Last 4096 bytes of stderr :
  0 in stage 2.0 (TID 119)
  19/04/26 14:27:37 INFO HadoopRDD: Input split: hdfs://datadev/data/dev/HIVE_SCHEMA/somedb.db/sbscr_usge_cycl_key_xref/000000_0_copy_2:0+6623042
  19/04/26 14:27:37 INFO OrcRawRecordMerger: min key = null, max key = null
  19/04/26 14:27:37 INFO ReaderImpl: Reading ORC rows from hdfs://datadev/data/dev/HIVE_SCHEMA/somedb.db/sbscr_usge_cycl_key_xref/000000_0_copy_2 with {include: [true, true, true], offset: 0, length: 9223372036854775807}
  19/04/26 14:29:00 ERROR Executor: Exception in task 5.0 in stage 2.0 (TID 119)
  java.lang.OutOfMemoryError
          at java.io.ByteArrayOutputStream.hugeCapacity(ByteArrayOutputStream.java:123)
          at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:117)
          at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
          at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
          at net.jpountz.lz4.LZ4BlockOutputStream.flushBufferedData(LZ4BlockOutputStream.java:205)
          at net.jpountz.lz4.LZ4BlockOutputStream.write(LZ4BlockOutputStream.java:158)
          at java.io.DataOutputStream.write(DataOutputStream.java:107)
          at org.apache.spark.sql.catalyst.expressions.UnsafeRow.writeToStream(UnsafeRow.java:554)
          at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:237)
          at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:228)
          at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:827)
          at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:827)
          at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
          at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
          at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
          at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
          at org.apache.spark.scheduler.Task.run(Task.scala:108)
          at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          at java.lang.Thread.run(Thread.java:748)
  19/04/26 14:29:00 ERROR SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[Executor task launch worker for task 119,5,main]
  java.lang.OutOfMemoryError
          at java.io.ByteArrayOutputStream.hugeCapacity(ByteArrayOutputStream.java:123)
          at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:117)
          at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
          at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
          at net.jpountz.lz4.LZ4BlockOutputStream.flushBufferedData(LZ4BlockOutputStream.java:205)
          at net.jpountz.lz4.LZ4BlockOutputStream.write(LZ4BlockOutputStream.java:158)
          at java.io.DataOutputStream.write(DataOutputStream.java:107)
          at org.apache.spark.sql.catalyst.expressions.UnsafeRow.writeToStream(UnsafeRow.java:554)
          at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:237)
          at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:228)
          at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:827)
          at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:827)
          at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
          at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
          at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
          at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
          at org.apache.spark.scheduler.Task.run(Task.scala:108)
          at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          at java.lang.Thread.run(Thread.java:748)
  19/04/26 14:29:00 INFO DiskBlockManager: Shutdown hook called
  19/04/26 14:29:00 INFO ShutdownHookManager: Shutdown hook called
  19/04/26 14:29:02 ERROR YarnScheduler: Lost executor 2 on worker03.some.com: Container marked as failed: container_e1223_1556277056929_0976_01_000003 on host: worker03.some.com. Exit status: 52. Diagnostics: Exception from container-launch.
  Container id: container_e1223_1556277056929_0976_01_000003
  Exit code: 52
  Shell output: main : command provided 1
  main : run as user is svc-bd-xdladmrw-dev
  main : requested yarn user is svc-bd-xdladmrw-dev
  Getting exit code file...
  Creating script paths...
  Writing pid file...
  Writing to tmp file /data/00/yarn/local/nmPrivate/application_1556277056929_0976/container_e1223_1556277056929_0976_01_000003/container_e1223_1556277056929_0976_01_000003.pid.tmp
  Writing to cgroup task files...
  Creating local dirs...
  Launching container...
  Getting exit code file...
  Creating script paths...
  
  
  Container exited with a non-zero exit code 52. Last 4096 bytes of stderr :
  0 in stage 2.0 (TID 119)
  19/04/26 14:27:37 INFO HadoopRDD: Input split: hdfs://datadev/data/dev/HIVE_SCHEMA/somedb.db/sbscr_usge_cycl_key_xref/000000_0_copy_2:0+6623042
  19/04/26 14:27:37 INFO OrcRawRecordMerger: min key = null, max key = null
  19/04/26 14:27:37 INFO ReaderImpl: Reading ORC rows from hdfs://datadev/data/dev/HIVE_SCHEMA/somedb.db/sbscr_usge_cycl_key_xref/000000_0_copy_2 with {include: [true, true, true], offset: 0, length: 9223372036854775807}
  19/04/26 14:29:00 ERROR Executor: Exception in task 5.0 in stage 2.0 (TID 119)
  java.lang.OutOfMemoryError
          at java.io.ByteArrayOutputStream.hugeCapacity(ByteArrayOutputStream.java:123)
          at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:117)
          at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
          at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
          at net.jpountz.lz4.LZ4BlockOutputStream.flushBufferedData(LZ4BlockOutputStream.java:205)
          at net.jpountz.lz4.LZ4BlockOutputStream.write(LZ4BlockOutputStream.java:158)
          at java.io.DataOutputStream.write(DataOutputStream.java:107)
          at org.apache.spark.sql.catalyst.expressions.UnsafeRow.writeToStream(UnsafeRow.java:554)
          at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:237)
          at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:228)
          at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:827)
          at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:827)
          at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
          at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
          at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
          at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
          at org.apache.spark.scheduler.Task.run(Task.scala:108)
          at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          at java.lang.Thread.run(Thread.java:748)
  19/04/26 14:29:00 ERROR SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[Executor task launch worker for task 119,5,main]
  java.lang.OutOfMemoryError
          at java.io.ByteArrayOutputStream.hugeCapacity(ByteArrayOutputStream.java:123)
          at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:117)
          at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
          at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
          at net.jpountz.lz4.LZ4BlockOutputStream.flushBufferedData(LZ4BlockOutputStream.java:205)
          at net.jpountz.lz4.LZ4BlockOutputStream.write(LZ4BlockOutputStream.java:158)
          at java.io.DataOutputStream.write(DataOutputStream.java:107)
          at org.apache.spark.sql.catalyst.expressions.UnsafeRow.writeToStream(UnsafeRow.java:554)
          at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:237)
          at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:228)
          at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:827)
          at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:827)
          at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
          at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
          at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
          at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
          at org.apache.spark.scheduler.Task.run(Task.scala:108)
          at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          at java.lang.Thread.run(Thread.java:748)
  19/04/26 14:29:00 INFO DiskBlockManager: Shutdown hook called
  19/04/26 14:29:00 INFO ShutdownHookManager: Shutdown hook called
  </code></pre>
  
  <p>Can anyone let me know if I am doing anything wrong here, like the memory allocation or something?, please suggest any alternatives to complete this job without getting the our of memory exception or worker node lost error. Any help or info is greatly appreciated.</p>
  
  <p>Thanks!</p>