title: EMR 5.x | Spark on Yarn | Exit code 137 and Java heap space Error
body: <p>I have been getting this error <code>Container exited with a non-zero exit code 137</code> while running spark on yarn. I have tried couple of techniques after going through but didnt help. The spark configurations looks like below:</p>
  
  <pre><code>spark.driver.memory 10G
  spark.driver.maxResultSize  2G
  spark.memory.fraction   0.8
  </code></pre>
  
  <p>I am using yarn in client mode. <code>spark-submit --packages com.databricks:spark-redshift_2.10:0.5.0 --jars RedshiftJDBC4-1.2.1.1001.jar elevatedailyjob.py &gt; log5.out 2&gt;&amp;1 &amp;</code></p>
  
  <p>The sample code :</p>
  
  <pre><code># Load the file (its a single file of 3.2GB)
  
  my_df = spark.read.csv('s3://bucket-name/path/file_additional.txt.gz', schema=MySchema, sep=';', header=True)
  
  # write the de_pulse_ip data into parquet format
  my_df = my_df.select("ip_start","ip_end","country_code","region_code","city_code","ip_start_int","ip_end_int","postal_code").repartition(50)
  my_df.write.parquet("s3://analyst-adhoc/elevate/tempData/de_pulse_ip1.parquet", mode = "overwrite")
  
  # read my_df data intp dataframe from parquet files
  my_df1 = spark.read.parquet("s3://bucket-name/path/my_df.parquet").repartition("ip_start_int","ip_end_int")
  
  #join with another dataset 200 MB
  my_df2 = my_df.join(my_df1, [my_df.ip_int_cast &gt; my_df1.ip_start_int,my_df.ip_int_cast &lt;= my_df1.ip_end_int], how='right')
  </code></pre>
  
  <p>Note: the input file is a single gzip file. It's unzipped size is 3.2GB</p>