title: How to run spark on kubernetes with limited resources for each namespace
body: <p>I have currently set up the spark standalone cluster in k8s using helm. It has a fixed number of workers (5 workers) and a single master. Currently doing spark-submit as below</p>
  <pre><code>spark-submit --class LogParser.LogBundleConfigFetcher --conf spark.submit.deployMode=cluster --conf spark.network.timeout=300 --conf spark.scheduler.mode=FAIR --conf **spark.master=spark://sm-care-alice-staging:7077** --conf spark.executor.cores=5 --conf spark.executor.memory=20g --conf spark.dynamicAllocation.maxExecutors=3 --conf spark.driver.memory=16g --conf spark.dynamicAllocation.enabled=false --conf spark.cores.max=15 http://minio.platform.svc.cluster.local:9000/alice-care/staging/config/spark/aliceparser.jar
  </code></pre>
  <p>But it has some limitations like we can not use cluster mode for pyspark- <strong>Cluster deploy mode is currently not supported for python applications on standalone clusters</strong></p>
  <p>So, I would like to use k8s as a cluster manager. But we have a huge k8s cluster and limited resources are allocated to each project (assigned by namespaces). Using k8s as cluster manager would spawn the pods dynamically so we can't control the resources (I know we can control the instances but in case of multiple spark-submit we have to restrict it to a limit of cores and memory).</p>
  <blockquote>
  </blockquote>
  <pre><code>spark-submit \
      --master k8s://https://$(k8s_ip):8443 \
      --deploy-mode cluster \
      --name spark-example \
      --conf spark.executor.instances=5 \
      --conf spark.kubernetes.container.image=pyspark-k8s-example:2.4.1 \
      --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
      --conf spark.kubernetes.pyspark.pythonVersion=3 \
      /usr/bin/run.py
  </code></pre>
  <p>I would like to know how can we control the k8s cluster manager to limit the resources i.e limit the cores to 100 irrespective of how many spark-submit being called.</p>
  <p>Or is there any other way that can run pyspark in cluster mode in the same setup?</p>