title: Failed to find data source: kafka. using Docker all-spark-notebook Spark Ver 3.1.1
body: <p>I'm very new to Spark and Kafka and I'm trying to run some sample code in Python (Jupyter) using Docker images downloaded, configured and executed with docker-compose.  I'm running Ubuntu 20.04 through Windows 10 WSL.</p>
  <p>Here are the steps I've followed so far:</p>
  <ol>
  <li><p>Using the docker-compose.yml file shown below, I run [docker-compose up -d]</p>
  </li>
  <li><p>All containers start successfully</p>
  </li>
  <li><p>I start a Jupyter notebook using <a href="http://127.0.0.1:8888/?token=" rel="nofollow noreferrer">http://127.0.0.1:8888/?token=</a></p>
  </li>
  <li><p>I execute the Python script line-by-line and it fails at</p>
  <pre><code>    .readStream \
      .format(&quot;kafka&quot;) \
      .option(&quot;kafka.bootstrap.servers&quot;, &quot;localhost:9092&quot;) \
      .option(&quot;subscribe&quot;, &quot;cloud&quot;) \
      .option(&quot;startingOffsets&quot;, &quot;earliest&quot;) \
      .load()```
  
  </code></pre>
  </li>
  </ol>
  <p>with the error:</p>
  <p><code>error preamble... AnalysisException: Failed to find data source: kafka. Please deploy the application as per the deployment section of &quot;Structured Streaming + Kafka Integration Guide&quot;.</code></p>
  <p>I'm clearly missing a step and I've searched in-vain for a solution.  Other answers to this problem suggest adding:</p>
  <p><code>import os os.environ['PYSPARK_SUBMIT_ARGS'] = '--packages org.apache.spark:spark-sql-kafka-0-10_2.12:3.1.1 pyspark-shell'</code></p>
  <p>which hasn't worked for me.  What step am I missing?</p>
  <p><strong>docker-compose.yml</strong></p>
  <pre><code>---
  version: '2'
  services:
    spark:
      image: jupyter/all-spark-notebook:latest
      ports:
        - &quot;8888:8888&quot;
      working_dir: /home/$USER/work
      volumes:
        - $PWD/work:/home/$USER/work
  
    zookeeper:
      image: confluentinc/cp-zookeeper:latest
      hostname: zookeeper
      container_name: zookeeper
      ports:
        - &quot;2181:2181&quot;
      environment:
        ZOOKEEPER_CLIENT_PORT: 2181
        ZOOKEEPER_TICK_TIME: 2000
        KAFKA_OPTS: &quot;-Dzookeeper.4lw.commands.whitelist=*&quot;
  
    kafka:
      image: confluentinc/cp-kafka:latest
      hostname: kafka
      container_name: kafka
      depends_on:
        - zookeeper
      ports:
        - &quot;9092:9092&quot;
        - &quot;29092:29092&quot;
      environment:
        KAFKA_BROKER_ID: 1
        KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
        KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
        KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
        KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
  
    connect:
      image: confluentinc/cp-kafka-connect:latest
      hostname: connect
      container_name: connect
      depends_on:
        - zookeeper
        - kafka
      ports:
        - &quot;8083:8083&quot;
      environment:
        CONNECT_BOOTSTRAP_SERVERS: kafka:9092
        CONNECT_REST_ADVERTISED_HOST_NAME: connect
        CONNECT_REST_PORT: 8083
        CONNECT_GROUP_ID: compose-connect-group
        CONNECT_CONFIG_STORAGE_TOPIC: docker-connect-configs
        CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: 1
        CONNECT_OFFSET_FLUSH_INTERVAL_MS: 10000
        CONNECT_OFFSET_STORAGE_TOPIC: docker-connect-offsets
        CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: 1
        CONNECT_STATUS_STORAGE_TOPIC: docker-connect-status
        CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: 1
        CONNECT_KEY_CONVERTER: org.apache.kafka.connect.json.JsonConverter
        CONNECT_VALUE_CONVERTER: org.apache.kafka.connect.json.JsonConverter
        CONNECT_INTERNAL_KEY_CONVERTER: org.apache.kafka.connect.json.JsonConverter
        CONNECT_INTERNAL_VALUE_CONVERTER: org.apache.kafka.connect.json.JsonConverter
        CONNECT_ZOOKEEPER_CONNECT: zookeeper:2181
        CONNECT_PLUGIN_PATH: /usr/share/java,/usr/share/confluent-hub-components,/data/connect-jars
        CONNECT_LOG4J_LOGGERS: org.apache.kafka.connect=DEBUG
      command:
        - bash
        - -c
        - |
          echo &quot;Installing connector plugins&quot;
          confluent-hub install --no-prompt confluentinc/kafka-connect-jdbc:latest
          cd /usr/share/confluent-hub-components/confluentinc-kafka-connect-jdbc/lib
          curl http://repo.odysseusinc.com/artifactory/community-libs-release-local/org/netezza/nzjdbc/1.0/nzjdbc-1.0.jar -o nzjdbc-1.0.jar
          /etc/confluent/docker/run &amp;
          sleep infinity
  </code></pre>
  <p><strong>Python Code</strong></p>
  <pre><code>#!/usr/bin/env python
  # coding: utf-8
  
  # In[ ]:
  
  
  import os
  os.environ['PYSPARK_SUBMIT_ARGS'] = '--packages org.apache.spark:spark-sql-kafka-0-10_2.12:3.1.1 pyspark-shell'
  
  
  # In[ ]:
  
  
  #    Spark
  import pyspark
  import findspark
  from pyspark import SparkContext
  from pyspark.sql import SparkSession
  #    Spark Streaming
  from pyspark.streaming import StreamingContext
  #    json parsing
  import json
  
  
  # In[ ]:
  
  
  findspark.init()
  
  
  # In[ ]:
  
  
  spark = SparkSession     .builder     .appName(&quot;My App&quot;)     .getOrCreate()
  #    .config(&quot;spark.jars&quot;, &quot;/path/to/jar.jar,/path/to/another/jar.jar&quot;) \
  
  
  # In[ ]:
  
  
  spark.version
  
  
  # In[ ]:
  
  
  df = spark         .readStream         .format(&quot;kafka&quot;)         .option(&quot;kafka.bootstrap.servers&quot;, &quot;localhost:9092&quot;)         .option(&quot;subscribe&quot;, &quot;cloud&quot;)         .option(&quot;startingOffsets&quot;, &quot;earliest&quot;)         .load()
  </code></pre>