title: Spark Structured Streaming foreachBatch and UPSERT (merge): to persist or not to persist?
body: <p>In case of stateful aggregation (arbitrary) in Structured Streaming with foreachBatch to merge update into delta table, should I persist batch dataframe inside foreachBatch before upserting or not?</p>
  
  <p>It seems for be that persist is not required since i'm writing to single data sink.</p>
  
  <p>On the other hand i have strong feeling that not persisting will cause source re-scan and trigger aggregation twice.</p>
  
  <p>Any comments/thoughts?</p>
  
  <pre><code>foreachBatch((VoidFunction2&lt;Dataset&lt;Row&gt;, Long&gt;) (batchDf, batchId) -&gt; 
          deltaTable.as("table").merge(batchDf.as("updates"), functions.expr("table.id=updates.id"))
          .whenNotMatched().insertAll()           // new session to be added
          .whenMatched()
          .updateAll()
          .execute())
  </code></pre>