title: Spark submit (2.3) on kubernetes cluster from Python
body: <p>So now that k8s is integrated directly with spark in 2.3 my spark submit from the console executes correctly on a kuberenetes master without any spark master pods running, spark handles all the k8s details:</p>
  
  <pre><code>spark-submit \
    --deploy-mode cluster \
    --class com.app.myApp \
    --master k8s://https://myCluster.com \
    --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
    --conf spark.app.name=myApp \
    --conf spark.executor.instances=10 \
    --conf spark.kubernetes.container.image=myImage \
    local:///myJar.jar
  </code></pre>
  
  <p>What I am trying to do is do a spark-submit via AWS lambda to my k8s cluster. Previously I used the command via the spark master REST API directly (without kubernetes):</p>
  
  <pre><code>request = requests.Request(
      'POST',
      "http://&lt;master-ip&gt;:6066/v1/submissions/create",
      data=json.dumps(parameters))
  prepared = request.prepare()
  session = requests.Session()
  response = session.send(prepared)
  </code></pre>
  
  <p>And it worked. Now I want to integrate Kubernetes and do it similarly where I submit an API request to my kubernetes cluster from python and have spark handle all the k8s details, ideally something like:</p>
  
  <pre><code>request = requests.Request(
      'POST',
      "k8s://https://myK8scluster.com:443",
      data=json.dumps(parameters))
  </code></pre>
  
  <p>Is it possible in the Spark 2.3/Kubernetes integration?</p>