title: Reusing pyspark cache and unpersist in for loop
body: <p>I have a lot of data that I'm trying to take out in chunks - let's say 3 chunks - and not have it all cached in memory at once. However, I'd like to save it (action) all at the same time afterwards.</p>
  <p>This is the current simplified strategy:</p>
  <pre><code>for query in [query1,query2,query3]:
  
      df = spark.sql(query)
  
      df.cache()
  
      df1 = df.filter('a')
      df2 = df.filter('b')
  
      final_output_1 = final_output_1.join(df1)
      final_output_2 = final_output_2.join(df2)
  
      df.unpersist()
  
  
  final_output_1.write.saveAsTable()
  final_output_2.write.saveAsTable()
  </code></pre>
  <p>So <strong>first question:</strong> would <code>unpersist()</code> not work here since there hasn't been an action yet on <code>df</code>?</p>
  <p><strong>second question:</strong> how does <code>df.cache()</code> work here when I'm reusing the <code>df</code> variable in the for loop? I know it's immutable so it would make a copy but would the <code>unpersist()</code> actually clear that memory?</p>