title: Spark YARN config issue : Container keep failing
body: <p>I am trying to save the data frame as textFile but it take much time even for small data. I believe there is something wrong with my config. Can some one tell me what I am doing wrong here ?</p>
  
  <pre><code>spark.default.parallelism   640
  spark.hadoop.fs.s3.cse.plaintextLength.enabled  false
  spark.hadoop.fs.s3n.filestatuscache.enable  true
  spark.hadoop.mapreduce.input.fileinputformat.split.maxsize  33554432
  spark.executor.id   driver
  spark.executor.instances    10
  spark.executor.memory   18g
  spark.executor.extraJavaOptions -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:MaxHeapFreeRatio=70 -XX:+CMSClassUnloadingEnabled -XX:OnOutOfMemoryError='kill -9 %p'
  
  spark.driver.memory 219695M
  spark.driver.port   51885
  spark.dynamicAllocation.enabled true
  spark.dynamicAllocation.initialExecutors    spark.dynamicAllocation.minExecutors
  spark.dynamicAllocation.maxExecutors    infinity
  spark.dynamicAllocation.minExecutors    120
  spark.dynamicAllocation.schedulerBacklogTimeout 1s
  spark.eventLog.enabled  true
  spark.executor.cores    32
  </code></pre>
  
  <p>I am having r3.8xLarge = 1 master with 10 slaves</p>
  
  <p>The error I am getting : </p>
  
  <pre><code>org.apache.spark.SparkException: Job aborted due to stage failure: Task 50 in stage 5.0 failed 4 times, most recent failure: Lost task 50.3 in stage 5.0 (TID 321, ip-172-31-3-183.ec2.internal): ExecutorLostFailure (executor 4 exited caused by one of the running tasks) Reason: Container marked as failed: container_1475709908651_0002_01_000005 on host: ip-172-31-3-183.ec2.internal. Exit status: 137. Diagnostics: Container killed on request. Exit code is 137 Container exited with a non-zero exit code 137 Killed by external signal Driver stacktrace:  
  at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1431) 
  at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1419)  
  at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1418) 
  at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) 
  at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) 
  at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1418) 
  at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799) 
  at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
  at scala.Option.foreach(Option.scala:236) 
  at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:799)  
  at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1640) 
  at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1599)  
  at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1588) 
  at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48) 
  at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:620)  
  at org.apache.spark.SparkContext.runJob(SparkContext.scala:1832) 
  at org.apache.spark.SparkContext.runJob(SparkContext.scala:1845) 
  at org.apache.spark.SparkContext.runJob(SparkContext.scala:1858)  
  at org.apache.spark.SparkContext.runJob(SparkContext.scala:1929)  
  at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:927) 
  at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
  at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)
  at org.apache.spark.rdd.RDD.withScope(RDD.scala:316) 
  at org.apache.spark.rdd.RDD.collect(RDD.scala:926) 
  at org.apache.spark.RangePartitioner$.sketch(Partitioner.scala:264) 
  at org.apache.spark.RangePartitioner.&lt;init&gt;(Partitioner.scala:126) 
  at org.apache.spark.rdd.OrderedRDDFunctions$$anonfun$sortByKey$1.apply(OrderedRDDFunctions.scala:62) 
  at org.apache.spark.rdd.OrderedRDDFunctions$$anonfun$sortByKey$1.apply(OrderedRDDFunctions.scala:61)  
  at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150) 
  at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111) 
  at org.apache.spark.rdd.RDD.withScope(RDD.scala:316) 
  at org.apache.spark.rdd.OrderedRDDFunctions.sortByKey(OrderedRDDFunctions.scala:61)
  at amazon.remo.spark.RemoPairRDDFunctions.saveAsRODB(RemoPairRDDFunctions.scala:62) 
  at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.&lt;init&gt;(&lt;console&gt;:192)
  at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.&lt;init&gt;(&lt;console&gt;:197) 
  at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.&lt;init&gt;(&lt;console&gt;:199)
  at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.&lt;init&gt;(&lt;console&gt;:201) 
  at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.&lt;init&gt;(&lt;console&gt;:203) 
  at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.&lt;init&gt;(&lt;console&gt;:205)
   at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.&lt;init&gt;(&lt;console&gt;:207) 
   at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.&lt;init&gt;(&lt;console&gt;:209) 
   at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.&lt;init&gt;(&lt;console&gt;:211) 
   at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.&lt;init&gt;(&lt;console&gt;:213) 
   at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.&lt;init&gt;(&lt;console&gt;:215) 
   at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.&lt;init&gt;(&lt;console&gt;:217) 
   at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.&lt;init&gt;(&lt;console&gt;:219)
    at $iwC$$iwC$$iwC$$iwC$$iwC.&lt;init&gt;(&lt;console&gt;:221) at $iwC$$iwC$$iwC$$iwC.&lt;init&gt;(&lt;console&gt;:223) at $iwC$$iwC$$iwC.&lt;init&gt;(&lt;console&gt;:225) at $iwC$$iwC.&lt;init&gt;(&lt;console&gt;:227) at $iwC.&lt;init&gt;(&lt;console&gt;:229) at &lt;init&gt;(&lt;console&gt;:231) at .&lt;init&gt;(&lt;console&gt;:235) 
   at .&lt;clinit&gt;(&lt;console&gt;)
    at .&lt;init&gt;(&lt;console&gt;:7) 
    at .&lt;clinit&gt;(&lt;console&gt;) 
    at $print(&lt;console&gt;) 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method
   ) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
   at java.lang.reflect.Method.invoke(Method.java:483) 
    at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)  
   at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1346)  
   at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840) 
    at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871) 
    at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819) 
    at org.apache.zeppelin.spark.SparkInterpreter.interpretInput(SparkInterpreter.java:664) 
    at org.apache.zeppelin.spark.SparkInterpreter.interpret(SparkInterpreter.java:629)
     at org.apache.zeppelin.spark.SparkInterpreter.interpret(SparkInterpreter.java:622) 
    at org.apache.zeppelin.interpreter.ClassloaderInterpreter.interpret(ClassloaderInterpreter.java:57) 
    at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:93) 
    at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:276)  
   at org.apache.zeppelin.scheduler.Job.run(Job.java:170) 
    at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:118)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
     at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) 
   at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) 
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745) 
  </code></pre>
  
  <p>I am using EMR 4.3.0 </p>