title: Why would spark app freeze when requesting new executors?
body: <p>New to Spark &amp; Yarn.  I'm running a spark app on Yarn (cloudera).  Everything works fine, but if I attempt to cache an RDD then the job freezes halfway through processing.</p>
  
  <p>The RDD is taking 3GB.  I'm trying to figure out how to troubleshoot this issue.  Clearly there must be a resource contention...</p>
  
  <pre><code>16/05/17 19:49:41 INFO ExecutorAllocationManager: Requesting 16 new executors because tasks are backlogged (new desired total will be 72)
  16/05/17 19:49:42 INFO ExecutorAllocationManager: Requesting 32 new executors because tasks are backlogged (new desired total will be 104)
  16/05/17 19:49:43 INFO ExecutorAllocationManager: Requesting 64 new executors because tasks are backlogged (new desired total will be 168)
  16/05/17 19:49:44 INFO ExecutorAllocationManager: Requesting 128 new executors because tasks are backlogged (new desired total will be 296)
  16/05/17 19:49:45 INFO ExecutorAllocationManager: Requesting 172 new executors because tasks are backlogged (new desired total will be 468)
  </code></pre>
  
  <p>...and it just hangs there...</p>