title: How can PySpark be called in debug mode?
body: <p>I have IntelliJ IDEA set up with Apache Spark 1.4.</p>
  
  <p>I want to be able to add debug points to my Spark Python scripts so that I can debug them easily.</p>
  
  <p>I am currently running this bit of Python to initialise the spark process</p>
  
  <pre><code>proc = subprocess.Popen([SPARK_SUBMIT_PATH, scriptFile, inputFile], shell=SHELL_OUTPUT, stdout=subprocess.PIPE)
  
  if VERBOSE:
      print proc.stdout.read()
      print proc.stderr.read()
  </code></pre>
  
  <p>When <code>spark-submit</code> eventually calls <code>myFirstSparkScript.py</code>, the debug mode is not engaged and it executes as normal. Unfortunately, editing the Apache Spark source code and running a customised copy is not an acceptable solution. </p>
  
  <p>Does anyone know if it is possible to have spark-submit call the Apache Spark script in debug mode? If so, how?</p>