title: How to get the output from console streaming sink in Zeppelin?
body: <p>I'm struggling to get the <code>console</code> sink working with <a href="https://spark.apache.org/docs/2.2.0/structured-streaming-programming-guide.html" rel="noreferrer">PySpark Structured Streaming</a> when run from Zeppelin. Basically, I'm not seeing any results printed to the screen, or to any logfiles I've found.</p>
  
  <p><strong>My question:</strong> Does anyone have a working example of using PySpark Structured Streaming with a sink that produces output visible in Apache Zeppelin? Ideally it would also use the socket source, as that's easy to test with.</p>
  
  <p>I'm using:</p>
  
  <ul>
  <li>Ubuntu 16.04</li>
  <li>spark-2.2.0-bin-hadoop2.7</li>
  <li>zeppelin-0.7.3-bin-all</li>
  <li>Python3</li>
  </ul>
  
  <p>I've based my code on the <a href="https://github.com/apache/spark/blob/master/examples/src/main/python/sql/streaming/structured_network_wordcount.py" rel="noreferrer">structured_network_wordcount.py example</a>. It works when run from the PySpark shell (<code>./bin/pyspark --master local[2]</code>); I see tables per batch.</p>
  
  <pre class="lang-py prettyprint-override"><code>%pyspark
  # structured streaming
  from pyspark.sql.functions import *
  lines = spark\
      .readStream\
      .format('socket')\
      .option('host', 'localhost')\
      .option('port', 9999)\
      .option('includeTimestamp', 'true')\
      .load()
  
  # Split the lines into words, retaining timestamps
  # split() splits each line into an array, and explode() turns the array into multiple rows
  words = lines.select(
      explode(split(lines.value, ' ')).alias('word'),
      lines.timestamp
  )
  
  # Group the data by window and word and compute the count of each group
  windowedCounts = words.groupBy(
      window(words.timestamp, '10 seconds', '1 seconds'),
      words.word
  ).count().orderBy('window')
  
  # Start running the query that prints the windowed word counts to the console
  query = windowedCounts\
      .writeStream\
      .outputMode('complete')\
      .format('console')\
      .option('truncate', 'false')\
      .start()
  
  print("Starting...")
  query.awaitTermination(20)
  </code></pre>
  
  <p>I'd expect to see printouts of results for each batch, but instead I just see <code>Starting...</code>, and then <code>False</code>, the return value of <code>query.awaitTermination(20)</code>.</p>
  
  <p>In a separate terminal I'm entering some data into a <code>nc -lk 9999</code> netcat session while the above is running.</p>