title: Spark' Dataset unpersist behaviour
body: <p>Recently I saw some strange behaviour of Spark.</p>
  
  <p>I have a pipeline in my application in which I'm manipulating one big Dataset - pseudocode:</p>
  
  <pre class="lang-scala prettyprint-override"><code>val data = spark.read (...)
  data.join(df1, "key") //etc, more transformations
  data.cache(); // used to not recalculate data after save
  data.write.parquet() // some save
  
  val extension = data.join (..) // more transformations - joins, selects, etc.
  extension.cache(); // again, cache to not double calculations
  extension.count();
  // (1)
  extension.write.csv() // some other save
  
  extension.groupBy("key").agg(some aggregations) //
  extension.write.parquet() // other save, without cache it will trigger recomputation of whole dataset
  </code></pre>
  
  <p>However when I call <code>data.unpersist()</code> i.e. in place <code>(1)</code>, Spark deletes from Storage all datasets, also the <code>extension</code> Dataset which is not the dataset I tried to unpersist.</p>
  
  <p>Is that an expected behaviour? How can I free some memory by <code>unpersist</code> on old Dataset without unpersisting all Dataset that was "next in chain"?</p>
  
  <p>My setup:</p>
  
  <ul>
  <li>Spark version: current master, RC for 2.3</li>
  <li>Scala: 2.11</li>
  <li>Java: OpenJDK 1.8</li>
  </ul>
  
  <p>Question looks similar to <a href="https://stackoverflow.com/questions/29903675/understanding-sparks-caching">Understanding Spark&#39;s caching</a>, but here I'm doing some actions before unpersist. At first I'm counting everything and then save into storage - I don't know if caching works the same in RDD like in Datasets</p>