title: how to run spark sql queries in every spark streaming batch?
body: <p>my scenario:</p>
  
  <ol>
  <li>Create a spark streaming context to pull data from kafka for batch time 50 sec</li>
  <li>create temp tables in spark for the data pulled.</li>
  <li>run spark sql query on 50s batch temp table data</li>
  </ol>
  
  <p>this must repeat till spark streaming job is running.</p>
  
  <p>but i am unable to figure out a way where spark sql will be run within the streaming context.</p>
  
  <p>As of now spark sql query is running only once and streaming is happening continuously.</p>
  
  <pre><code>val nv = (new Streamreader(kafkaParams,db))
  
    kafkaOpTopic.split(",").foreach(x =&gt; {
      nv.streader(ssc, Array(x))
    })
    spark.sql("show tables").show()
    ssc.start()
    ssc.awaitTermination()
  </code></pre>
  
  <p>StreamReader is doing kafka pull operation.</p>
  
  <p>What would be the right way to run spark sql at the end of every spark streaming batch window ? how to realize it ?</p>
  
  <p>Any idea would be greatly helpful.</p>