title: PySpark Kafka py4j.protocol.Py4JJavaError: An error occurred while calling o28.load
body: <p>While converting Kafka messages to dataframe am getting error while passing the packages as an argument.</p>
  
  <pre><code>from pyspark.sql import SparkSession, Row
  from pyspark.context import SparkContext
  from kafka import KafkaConsumer
  import os
  
  os.environ['PYSPARK_SUBMIT_ARGS'] = '--jars spark-sql-kafka-0-10_2.11-2.0.2.jar,spark-streaming-kafka-0-8-assembly_2.11-2.3.1.jar pyspark-shell'
  
  sc = SparkContext.getOrCreate()
  spark = SparkSession(sc)
  
  df = spark \
    .read \
    .format("kafka") \
    .option("kafka.bootstrap.servers", "localhost:9092") \
    .option("subscribe", "Jim_Topic") \
    .load()
  df.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)")
  </code></pre>
  
  <blockquote>
    <p>py4j.protocol.Py4JJavaError: An error occurred while calling o28.load.
    : java.util.ServiceConfigurationError: org.apache.spark.sql.sources.DataSourceRegister: Provider org.apache.spark.sql.kafka010.KafkaSourceProvider could not be instantiated</p>
  </blockquote>