title: Spark/k8s: How to run spark submit on Kubernetes with client mode
body: <p>I am trying to use <code>spark-submit</code> with <code>client</code> mode in the kubernetes pod to submit jobs to EMR (Due to some other infra issues, we don't allow <code>cluster</code> mode). 
  By default, <code>spark-submit</code> uses the <code>hostname</code> of the pod as the <code>spark.driver.host</code> and the <code>hostname</code> is the pod's hostname so <code>spark executor</code> could not resolve it. And the <code>spark.driver.port</code> is also locally to the pod (container). </p>
  
  <p>I know a way to pass some confs to <code>spark-submit</code> so that the <code>spark executor</code> can talk to the <code>driver</code>, those configs are: </p>
  
  <p><code>--conf spark.driver.bindAddress=0.0.0.0 --conf spark.driver.host=$HOST_IP_OF_K8S_WORKER --conf spark.driver.port=32000 --conf spark.driver.blockManager.port=32001</code></p>
  
  <p>and create a service to in the kubernetes so that <code>spark executor</code> can talk to the <code>driver</code>:</p>
  
  <pre><code>apiVersion: v1
  kind: Service
  metadata:
    name: spark-block-manager
    namespace: my-app
  spec:
    selector:
      app: my-app
    type: NodePort
    ports:
      - name: port-0
        nodePort: 32000
        port: 32000
        protocol: TCP
        targetPort: 32000
      - name: port-1
        nodePort: 32001
        port: 32001
        protocol: TCP
        targetPort: 32001
      - name: port-2
        nodePort: 32002
        port: 32002
        protocol: TCP
        targetPort: 32002
  </code></pre>
  
  <hr>
  
  <p>But the issue is there are can be more than 1 pods running on one k8s worker and even more than 1 <code>spark-submit</code> jobs in one pod. So before launching a pod, we need to dynamically select few available ports in the k8s node and create a service to do the port mapping and then during launching the pod, pass those ports into the pod to tell <code>spark-submit</code> to use them. I feel like this is a little bit complex.</p>
  
  <p>Using <code>hostNetwork: true</code> could potentially solve this issue but it introduces lots of other issues in our infra so this is not an option.</p>
  
  <p>If <code>spark-submit</code> can support <code>bindPort</code> concept just like <code>driver.bindAddress</code> and <code>driver.host</code> or support <code>proxy</code>, it will be cleaner to solve the issue.</p>
  
  <p>Does someone have similar situation? Please share some insights.</p>
  
  <p>Thanks.</p>
  
  <p>Additional context:
  <code>spark version</code>: <code>2.4</code></p>