title: How to tune the spark application in order to avoid OOM exception
body: <p>I use Spark 2.0.2.</p>
  
  <p>I am trying to run a spark application that does prediction on an already created model.</p>
  
  <p>Cluster Info: m4.2xlarge 16 vCPU, 32 GiB memory, EBS only storage EBS Storage:1000 GiB</p>
  
  <p>Based on the recommendations made <a href="http://blog.cloudera.com/blog/2015/03/how-to-tune-your-apache-spark-jobs-part-2/" rel="nofollow noreferrer">here</a> I have made a <a href="https://docs.google.com/spreadsheets/d/1_bOmD3VxX6fKT1yM_wsU_cdgLD_HYtXzRRxOlPuxMvc/edit?usp=sharing" rel="nofollow noreferrer">Google-Spreadsheet</a> to compute the tuning parameters. </p>
  
  <p>No matter what I try I am getting the following 2 exceptions:</p>
  
  <ol>
  <li>Container killed by YARN for exceeding memory limits. 10.0 GB of 10 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.</li>
  <li>Executor heartbeat timed out after 159126 ms </li>
  </ol>
  
  <p>Below is the code that I am trying to execute</p>
  
  <pre><code>val allGears = sc.textFile(allGearsFilePath)
  val allUsers = sc.textFile(allUsersFilePath)
  val allUserItems = allUsers.cartesian(allGears).map{ case(x,y) =&gt; (x.toInt, y.toInt)}
  allUserItems.cache()
  
  
  val gearPurchased = sc.textFile(gearPurchaseRating)
  val gearAddedToCart = sc.textFile(gearAddToCartRating)
  val gearShoppingUserToItem = gearPurchased.map(_.split(',') match   { case Array(user, item, rate) =&gt; (user.toInt, item.toInt) })
  gearShoppingUserToItem.cache()
  val allUserItemToGearShoppingUnion = allUserItems.union(gearShoppingUserToItem)
  val allUserItemToGearShoppingIntersection = allUserItems.intersection(gearShoppingUserToItem)
  val FinalSubtraction = allUserItems.subtract(gearShoppingUserToItem)
  val nonPurchasedGears = FinalSubtraction
  nonPurchasedGears.cache()
  allUserItems.unpersist()
  gearShoppingUserToItem.unpersist()
  val out = model.predict(nonPurchasedGears)
  </code></pre>
  
  <p>I get the exception when I try to Predict what gears can be purchased by the users.</p>
  
  <p>Below is the spark-submit command that I run </p>
  
  <pre><code>spark-submit --jars jedis-2.7.2.jar,commons-pool2-2.3.jar,spark-redis-0.3.2.jar,SparkHBase.jar,recommendcontentslib_2.11-1.0.jar --class org.digitaljuice.itemrecommender.RecommendGears --master yarn --driver-memory 2g --num-executors 5 --executor-memory 9g --executor-cores 5 --conf spark.yarn.executor.memoryOverhead=1024 recommendersystem_2.11-0.0.1.jar /work/output/gearpurchaserating/part-00000 /work/output/gearaddtocartrating/part-00000 /work/output/allGears/part-00000 /work/output/allAccounts/part-00000 /work/allaccounts/acc_toacc/part-m-00000 /work/Recommendations/ /work/TrainingModel
  </code></pre>
  
  <p>How can I tune the application so that it runs and makes the prediction?
  I have tried various things, but nothing seems to work, so I am guessing I am not tuning the application right. Kindly help.</p>
  
  <p>Thanks</p>