title: Spark Using Disk Resources When Memory is Available
body: <p>I'm working on optimizing the performance of my Spark cluster (run on AWS EMR) that is performing Collaborative Filtering using the ALS matrix factorization algorithm.  We are using quite a few factors and iterations so I'm trying to optimize these steps in particular.  I am trying to understand why I am using disk space when I have plenty of memory available. Here is the total cluster memory available:</p>
  
  <p><a href="https://i.stack.imgur.com/HAsvF.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/HAsvF.png" alt="enter image description here"></a></p>
  
  <p>Here is the remaining disk space (notice the dips of disk utilization):</p>
  
  <p><a href="https://i.stack.imgur.com/rDa7V.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/rDa7V.png" alt="enter image description here"></a></p>
  
  <p>I've tried looking at the Yarn manager and it looks like it shows that each node slave has: 110 GB (used)   4 GB (avail.).  You can also see the total allocated on the first image (700 GB).  I've also tried changing the ALS source and forcing the <code>intermediateRDDStorageLevel</code> and <code>finalRDDStorageLevel</code> from MEMORY_AND_DISK to MEMORY_ONLY and that didn't affect anything.</p>
  
  <p>I am not persisting my RDD's anywhere else in my code so I'm not sure where this disk utilization is coming from.  I'd like to better utilize the resources on my cluster, any ideas?  How can I more effectively use the available memory?</p>