title: How to configure memory usage correctly for small job in pyspark?
body: <p>I am getting OOMs with a fairly small job while running <code>pyspark</code> on <code>YARN</code>.  </p>
  
  <p>The job DOES get picked up by spark and yarn both, but then shortly after it fails with an OOM <code>Exit status: 52</code>:  </p>
  
  <pre><code>ERROR cluster.YarnScheduler: Lost executor 1 on &lt;ip&gt;: Container marked as failed: containerID 2 on host: &lt;ip&gt;. Exit status: 52. Diagnostics: Exception from container-launch.
  </code></pre>
  
  <p>And when I check the yarn log files for this application I see the following:  </p>
  
  <pre><code>    date 19:03:19 INFO hadoop.ColumnChunkPageWriteStore: written 10,852B for [user] BINARY: 11,052 values, 11,764B raw, 10,786B comp, 1 pages, encodings: [BIT_PACKED, PLAIN_DICTIONARY, RLE], dic { 3,607 entries, 78,442B raw, 3,607B comp}
  date 19:03:20 INFO mapred.SparkHadoopMapRedUtil: attempt_date1903_0006_m_000013_0: Committed
  date 19:03:20 INFO executor.Executor: Finished task 13.0 in stage 6.0 (TID 1058). 2077 bytes result sent to driver
  date time:07 INFO executor.Executor: Executor is trying to kill task 70.0 in stage 6.0 (TID 1115)
  date time:07 INFO executor.Executor: Executor is trying to kill task 127.0 in stage 6.0 (TID 1170)
  Traceback (most recent call last):
    File "/usr/lib/spark/python/pyspark/daemon.py", line 157, in manager
      code = worker(sock)
    File "/usr/lib/spark/python/pyspark/daemon.py", line 61, in worker
      worker_main(infile, outfile)
    File "/usr/lib/spark/python/pyspark/worker.py", line 136, in main
      if read_int(infile) == SpecialLengths.END_OF_STREAM:
    File "/usr/lib/spark/python/pyspark/serializers.py", line 545, in read_int
      raise EOFError
  EOFError
  date time:07 INFO executor.CoarseGrainedExecutorBackend: Driver commanded a shutdown
  </code></pre>
  
  <p>I am not sure what is causing this issue, since it seems like it is having trouble with the files but the spark driver memory is 8GB.</p>
  
  <p>machines look like this:</p>
  
  <pre><code>21 nodes, 64GB each, 8 cores each
  </code></pre>
  
  <p>spark-defaults.conf is:</p>
  
  <pre><code>spark.executor.memory=30928mb
  spark.driver.memory 8g
  spark.executor.instances=30
  spark.executor.cores = 7
  spark.yarn.executor.memoryOverhead = 19647
  </code></pre>
  
  <p>yarn-site.xml is:</p>
  
  <pre><code>yarn.scheduler.maximum-allocation-vcores = 1024
  yarn.scheduler.maximum-allocation-mb = 61430
  yarn.nodemanager.resource.memory-mb = 61430
  yarn.nodemanager.resource.cpu-vcores = 7 (left one for the driver)
  </code></pre>
  
  <p>Am I configuring anything incorrectly here? </p>
  
  <p>Also, the spark UI is showing: </p>
  
  <pre><code>URL: spark://ip:7077
  REST URL: spark://ip:6066
  Alive Workers: 30
  Cores in use: 240 Total, 0 Used
  Memory in use: 1769.7 GB Total, 0.0 B Used
  Applications: 0 Running, 0 Completed
  Drivers: 0 Running, 0 Completed
  Status: ALIVE
  </code></pre>
  
  <p>Any reason why the container would require more memory?  I am not sure whetehr to increase or decreases memory?</p>