title: spark container get killed by yarn
body: <p>I have a huge dataset of 675GB parquet file with snappy compression and i have to join it with 4 , 5 tables with size like 10 GB . I have a cluster of 500+ nodes each having 128GB ram, but i can only run a executor atmost 28 GB otherwise yarn does not allocate memory.
  Please advice how should i procced with this scenario.
  Currently i am running pyspark 1.6 and i am only runnning 1 executor per node with 26 Gb ram.
  But if i am running the whole join in hive it takes time but get completes.
  How should i use my cluster effeciently and procces with this join in spark  </p>
  
  <p>Thanks
  sPradeep</p>