title: Why would persisting RDD to DISK NOT improve performance?
body: <p>I have the following DAG</p>
  
  <ol>
  <li>Generate an RDD with 200 million records</li>
  <li>Persist the RDD to disk (StorageLevel.DISK_ONLY())</li>
  <li>Filter every even record in the RDD (50% of the records)</li>
  <li>Store to Disk (<code>saveAsNewAPIHadoopDataset</code>)</li>
  <li>Filter every odd record in the RDD (50% of the records)</li>
  <li>Store to Disk (<code>saveAsNewAPIHadoopDataset</code>)</li>
  </ol>
  
  <p>The size of the RDD on disk is <strong>100GB</strong></p>
  
  <p>The total number of time taken to accomplish the first action (step 4) is <strong>10 minutes</strong></p>
  
  <p>I would have thought that for the second action (step 6) the RDD would be loaded from disk filtered, etc.  Taking much shorter time.</p>
  
  <p>But in fact it takes the same amount of time <strong>10 minutes</strong>!  </p>
  
  <p>Would loading the 100GB rdd from disk really cost the same amount of time as generating the rdd?</p>
  
  <p>What could explain this?  Am I hitting some sort of IO bottleneck?  How can troubleshoot this?</p>