title: Submitting pyspark script to a remote Spark server?
body: <p>This is probably a really silly question, but I can't find the answer with Google. I've written a simple pyspark ETL script that reads in a CSV and writes it to Parquet, something like this:</p>
  
  <pre><code>spark = SparkSession.builder.getOrCreate()
  sqlContext = SQLContext(spark.sparkContext)
  df = sqlContext.read.csv(input_filename)
  df.write.parquet(output_path)
  </code></pre>
  
  <p>To run it, I start up a local Spark cluster in Docker:</p>
  
  <pre><code>$ docker run --network=host jupyter/pyspark-notebook
  </code></pre>
  
  <p>I run the Python script and it connects to this local Spark cluster and all works as expected.</p>
  
  <p>Now I'd like to run the same script on a remote Spark cluster (AWS EMR). Can I just specify a remote IP address somewhere when initialising the Spark context? Or am I misunderstanding how Spark works?</p>