title: Cannot run spark submit in standalone spark cluster
body: <p>I am working with the following docker-compose image to build a spark standalone cluster:</p>
  <pre><code>---
  # ----------------------------------------------------------------------------------------
  # -- Docs: https://github.com/cluster-apps-on-docker/spark-standalone-cluster-on-docker --
  # ----------------------------------------------------------------------------------------
  version: &quot;3.6&quot;
  volumes:
    shared-workspace:
      name: &quot;hadoop-distributed-file-system&quot;
      driver: local
  services:
    jupyterlab:
      image: andreper/jupyterlab:3.0.0-spark-3.0.0
      container_name: jupyterlab
      ports:
        - 8888:8888
        - 4040:4040
      volumes:
        - shared-workspace:/opt/workspace
    spark-master:
      image: andreper/spark-master:3.0.0
      container_name: spark-master
      ports:
        - 8080:8080
        - 7077:7077
      volumes:
        - shared-workspace:/opt/workspace
    spark-worker-1:
      image: andreper/spark-worker:3.0.0
      container_name: spark-worker-1
      environment:
        - SPARK_WORKER_CORES=1
        - SPARK_WORKER_MEMORY=512m
      ports:
        - 8081:8081
      volumes:
        - shared-workspace:/opt/workspace
      depends_on:
        - spark-master
    spark-worker-2:
      image: andreper/spark-worker:3.0.0
      container_name: spark-worker-2
      environment:
        - SPARK_WORKER_CORES=1
        - SPARK_WORKER_MEMORY=512m
      ports:
        - 8082:8081
      volumes:
        - shared-workspace:/opt/workspace
      depends_on:
        - spark-master
  </code></pre>
  <p>I followed this guide: <a href="https://towardsdatascience.com/apache-spark-cluster-on-docker-ft-a-juyterlab-interface-418383c95445" rel="nofollow noreferrer">https://towardsdatascience.com/apache-spark-cluster-on-docker-ft-a-juyterlab-interface-418383c95445</a>.</p>
  <p>Here can be found the Github repo: <a href="https://github.com/cluster-apps-on-docker/spark-standalone-cluster-on-docker" rel="nofollow noreferrer">https://github.com/cluster-apps-on-docker/spark-standalone-cluster-on-docker</a></p>
  <p>I can run the cluster and I can run code inside of the jupyter container, connecting to the master spark node without problems.</p>
  <p>The problem starts when I want to run the spark code with spark submit. I really cannot understand how the cluster works. When I run inside the Jupyter container, I can quickly see where the scripts I create are, but I can't find them in the spark master container. If I check the docker-compose.yml, the volumes indicates that the folder where the scripts are stored is:</p>
  <pre><code> volumes:
        - shared-workspace:/opt/workspace
  </code></pre>
  <p>But I cannot find this folder in any of the spark containers.</p>
  <p>When I run, spark submit, I run it once I have executed inside of the Jupyter container. In the Jupyter container I have all the scripts that I am working with, but I have the doubt when I write the following command: <code>spark submit --master spark:// spark-master:7077 &lt;PATH to my python script&gt;</code>, the path of the python script, is the path where the script in Jupyter container or spark master container?</p>
  <p>I can run the <code>spark submit</code> command without specifying the master, then it runs locally, and it runs without problems inside of the Jupyter container.</p>
  <p>This is the python code I am executing:</p>
  <pre><code>from pyspark.sql import SparkSession
  from pyspark import SparkContext, SparkConf
  from os.path import expanduser, join, abspath
  
  sparkConf = SparkConf()
  sparkConf.setMaster(&quot;spark://spark-master:7077&quot;)
  sparkConf.setAppName(&quot;pyspark-4&quot;)
  sparkConf.set(&quot;spark.executor.memory&quot;, &quot;2g&quot;)
  sparkConf.set(&quot;spark.driver.memory&quot;, &quot;2g&quot;)
  sparkConf.set(&quot;spark.executor.cores&quot;, &quot;1&quot;)
  sparkConf.set(&quot;spark.driver.cores&quot;, &quot;1&quot;)
  sparkConf.set(&quot;spark.dynamicAllocation.enabled&quot;, &quot;false&quot;)
  sparkConf.set(&quot;spark.shuffle.service.enabled&quot;, &quot;false&quot;)
  sparkConf.set(&quot;spark.sql.warehouse.dir&quot;, warehouse_location)
  spark = SparkSession.builder.config(conf=sparkConf).getOrCreate()
  sc = spark.sparkContext
  
  df = spark.createDataFrame(
      [
          (1, &quot;foo&quot;),  # create your data here, be consistent in the types.
          (2, &quot;bar&quot;),
      ],
      [&quot;id&quot;, &quot;label&quot;],  # add your column names here
  )
  print(df.show())
  </code></pre>
  <p>But when I specify the <code>master= --master spark:// spark-master: 7077</code>, and specifying the path where the script lives in the jupyter container:</p>
  <pre><code>spark-submit --master spark://spark-master:7077 test.py
  
  </code></pre>
  <p>ant this are the logs I receive:</p>
  <pre><code>21/06/06 21:32:04 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
  21/06/06 21:32:08 INFO SparkContext: Running Spark version 3.0.0
  21/06/06 21:32:09 INFO ResourceUtils: ==============================================================
  21/06/06 21:32:09 INFO ResourceUtils: Resources for spark.driver:
  
  21/06/06 21:32:09 INFO ResourceUtils: ==============================================================
  21/06/06 21:32:09 INFO SparkContext: Submitted application: pyspark-4
  21/06/06 21:32:09 INFO SecurityManager: Changing view acls to: root
  21/06/06 21:32:09 INFO SecurityManager: Changing modify acls to: root
  21/06/06 21:32:09 INFO SecurityManager: Changing view acls groups to: 
  21/06/06 21:32:09 INFO SecurityManager: Changing modify acls groups to: 
  21/06/06 21:32:09 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
  21/06/06 21:32:12 INFO Utils: Successfully started service 'sparkDriver' on port 45627.
  21/06/06 21:32:12 INFO SparkEnv: Registering MapOutputTracker
  21/06/06 21:32:13 INFO SparkEnv: Registering BlockManagerMaster
  21/06/06 21:32:13 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
  21/06/06 21:32:13 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
  21/06/06 21:32:13 INFO SparkEnv: Registering BlockManagerMasterHeartbeat
  21/06/06 21:32:13 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-5a81855c-3160-49a5-b9f9-9cdfe6e5ca62
  21/06/06 21:32:14 INFO MemoryStore: MemoryStore started with capacity 366.3 MiB
  21/06/06 21:32:14 INFO SparkEnv: Registering OutputCommitCoordinator
  21/06/06 21:32:16 INFO Utils: Successfully started service 'SparkUI' on port 4040.
  21/06/06 21:32:16 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://3b232f9ed93b:4040
  21/06/06 21:32:19 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://spark-master:7077...
  21/06/06 21:32:20 INFO TransportClientFactory: Successfully created connection to spark-master/172.21.0.5:7077 after 284 ms (0 ms spent in bootstraps)
  21/06/06 21:32:23 INFO StandaloneSchedulerBackend: Connected to Spark cluster with app ID app-20210606213223-0000
  21/06/06 21:32:23 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 46539.
  21/06/06 21:32:23 INFO NettyBlockTransferService: Server created on 3b232f9ed93b:46539
  21/06/06 21:32:23 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
  21/06/06 21:32:23 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 3b232f9ed93b, 46539, None)
  21/06/06 21:32:23 INFO BlockManagerMasterEndpoint: Registering block manager 3b232f9ed93b:46539 with 366.3 MiB RAM, BlockManagerId(driver, 3b232f9ed93b, 46539, None)
  21/06/06 21:32:23 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 3b232f9ed93b, 46539, None)
  21/06/06 21:32:23 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 3b232f9ed93b, 46539, None)
  21/06/06 21:32:25 INFO StandaloneSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
  21/06/06 21:32:29 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('/opt/workspace/spark-warehouse').
  21/06/06 21:32:29 INFO SharedState: Warehouse path is '/opt/workspace/spark-warehouse'.
  ESTOY AQUI¿¿
  21/06/06 21:33:09 INFO CodeGenerator: Code generated in 1925.0009 ms
  21/06/06 21:33:09 INFO SparkContext: Starting job: showString at NativeMethodAccessorImpl.java:0
  21/06/06 21:33:09 INFO DAGScheduler: Got job 0 (showString at NativeMethodAccessorImpl.java:0) with 1 output partitions
  21/06/06 21:33:09 INFO DAGScheduler: Final stage: ResultStage 0 (showString at NativeMethodAccessorImpl.java:0)
  21/06/06 21:33:09 INFO DAGScheduler: Parents of final stage: List()
  21/06/06 21:33:09 INFO DAGScheduler: Missing parents: List()
  21/06/06 21:33:10 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[6] at showString at NativeMethodAccessorImpl.java:0), which has no missing parents
  21/06/06 21:33:10 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 11.3 KiB, free 366.3 MiB)
  21/06/06 21:33:11 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 5.9 KiB, free 366.3 MiB)
  21/06/06 21:33:11 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 3b232f9ed93b:46539 (size: 5.9 KiB, free: 366.3 MiB)
  21/06/06 21:33:11 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1200
  21/06/06 21:33:11 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 0 (MapPartitionsRDD[6] at showString at NativeMethodAccessorImpl.java:0) (first 15 tasks are for partitions Vector(0))
  21/06/06 21:33:11 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
  21/06/06 21:33:26 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
  21/06/06 21:33:41 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
  21/06/06 21:33:56 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
  21/06/06 21:34:11 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
  21/06/06 21:34:26 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
  </code></pre>
  <p>When I execute the same code, inside of a jupyter notebook, it works without problems.</p>
  <p>It is because the path that I have to indicate for the script, is the path where the script lives in the spark-master node? or I am confounding things here</p>