title: kafka to pyspark structured streaming, parsing json as dataframe
body: <p>I am experimenting with spark structured streaming (spark v2.2.0) to consume json data from kafka. However I encountered the following error.</p>
  
  <blockquote>
    <p>pyspark.sql.utils.StreamingQueryException: 'Missing required
    configuration "partition.assignment.strategy" which has no default
    value.</p>
  </blockquote>
  
  <p>Does anyone know why? The job was submitted using spark-submit below.</p>
  
  <pre><code>spark-submit --packages org.apache.spark:spark-sql-kafka-0-10_2.11:2.2.0 sparksstream.py
  </code></pre>
  
  <p>This is the entire python script.</p>
  
  <pre><code>from pyspark.sql import SparkSession
  from pyspark.sql.functions import *
  from pyspark.sql.types import *
  
  spark = SparkSession \
      .builder \
      .appName("test") \
      .getOrCreate()
  
  # Define schema of json
  schema = StructType() \
          .add("Session-Id", StringType()) \
          .add("TransactionTimestamp", IntegerType()) \
          .add("User-Name", StringType()) \
          .add("ID", StringType()) \
          .add("Timestamp", IntegerType())
  
  # load data into spark-structured streaming
  df = spark \
        .readStream \
        .format("kafka") \
        .option("kafka.bootstrap.servers", "xxxx:9092") \
        .option("subscribe", "topicName") \
        .load() \
        .select(from_json(col("value").cast("string"), schema).alias("parsed_value"))
  
  # Print output
  query = df.writeStream \
              .outputMode("append") \
              .format("console") \
              .start()
  </code></pre>