title: Running chained queries using structured streaming (PySpark)
body: <p>My code is something like</p>
  
  <pre><code>df = spark.readStream.option("header","true") \
      .schema(df_schema)\
      .csv(df_file)
  df2 = df.filter(df.col == 1)
  df3 = df2.withColumn("new_col", udf_f(df2.some_col))
  dfc = df3.where(df3.new_col == 2).count()
  query = dfc.writeStream.outputMode("append").format("console").start()
  query.awaitTermination()
  </code></pre>
  
  <p>I get the error message <code>Queries with streaming sources must be executed with writeStream.start()</code> at the <code>dfc</code> line but I'm not sure what I'm doing wrong. Does Spark structured streaming not support chained queries like this? I'm not doing any branches as per my knowledge.</p>
  
  <p>EDIT:</p>
  
  <p>By removing <code>count()</code> from the <code>dfc</code> line, I got a new error <code>StreamingQueryException: Exception thrown in awaitResult</code> arising from the <code>query.awaitTermination()</code> call. Any idea why <code>count()</code> did not work and also why the new error arose? </p>
  
  <p>EDIT 2:</p>
  
  <p>If I log to console directly without running all the intermediate queries after df, it works. However, every time I try to run an additional query, the <code>StreamingQueryException</code> is raised.</p>