title: Spark access data frame from outside foreach batch (Strcutred Streaming)
body: <p>I want to create and update a data frame inside the foreach batch of a spark stream and access it outside the foreach batch iterator below is what I am trying to do in spark structured streaming.
  Is it possible to access data frames which are created or updated inside foreach batch from outside for each batch in spark structured streaming ?</p>
  <pre><code>// assign a empty data frame
  var df1: Option[DataFrame] = None: Option[DataFrame]
  
  validatedFinalDf.writeStream
    .foreachBatch { (batchDF: DataFrame, batchId: Long) =&gt;
      println(&quot;I am here printing batchDF&quot;)
      batchDF.withColumn(&quot;extra&quot;, lit(&quot;batch-df&quot;)).show()
  
      // un presist the data frame if it has data
      if (df1 != None) {
        df1.get.unpersist()
      }
  
      // assign data to data frame
      df1 = Some(batchDF.withColumn(&quot;extra&quot;, lit(&quot;batch-df-dim&quot;)))
  }.start()
  
  // access data frame outside foreach not working stale data ....
  if (df1 != None) {
        df1.get.show()
  }
  
  spark.streams.awaitAnyTermination()
  
  </code></pre>
  <p>I cant even access temp tables which are created inside foreach batch from outside for each batch.
  Even the data frame which is updated inside foreach batch shows stale data from outside foreach batch.</p>
  <p>Thanks
  Sri</p>