title: Access to Spark from Flask app
body: <p>I wrote a simple Flask app to pass some data to Spark.  The script works in IPython Notebook, but not when I try to run it in it's own server.  I don't think that the Spark context is running within the script.  How do I get Spark working in the following example?</p>
  
  <pre><code>from flask import Flask, request
  from pyspark import SparkConf, SparkContext
  
  app = Flask(__name__)
  
  conf = SparkConf()
  conf.setMaster("local")
  conf.setAppName("SparkContext1")
  conf.set("spark.executor.memory", "1g")
  sc = SparkContext(conf=conf)
  
  @app.route('/accessFunction', methods=['POST'])
  def toyFunction():
      posted_data = sc.parallelize([request.get_data()])
      return str(posted_data.collect()[0])
  
  if __name__ == '__main_':
      app.run(port=8080)
  </code></pre>
  
  <p>In IPython Notebook I don't define the <code>SparkContext</code> because it is automatically configured.  I don't remember how I did this, I followed some blogs. </p>
  
  <p>On the Linux server I have set the .py to always be running and installed the latest Spark by following up to step 5 of <a href="https://districtdatalabs.silvrback.com/getting-started-with-spark-in-python" rel="noreferrer">this guide</a>.</p>
  
  <p><strong>Edit</strong>:</p>
  
  <p>Following the advice by davidism I have now instead resorted to simple programs with increasing complexity to localise the error. </p>
  
  <p>Firstly I created .py with just the script from the answer below (after appropriately adjusting the links):</p>
  
  <pre><code>import sys
  try:
      sys.path.append("your/spark/home/python")
      from pyspark import context
      print ("Successfully imported Spark Modules")
  except ImportError as e:
      print ("Can not import Spark Modules", e)
  </code></pre>
  
  <p>This returns "Successfully imported Spark Modules". However, the next .py file I made returns an exception:</p>
  
  <pre><code>from pyspark import SparkContext
  sc = SparkContext('local')
  rdd = sc.parallelize([0])
  print rdd.count()
  </code></pre>
  
  <p>This returns exception:</p>
  
  <p>"Java gateway process exited before sending the driver its port number"</p>
  
  <p>Searching around for similar problems I found <a href="https://forums.databricks.com/questions/1662/spark-python-java-gateway-process-exited-before-se.html" rel="noreferrer">this page</a> but when I run this code nothing happens, no print on the console and no error messages. Similarly, <a href="http://ambracode.com/index/show/153755" rel="noreferrer">this</a> did not help either, I get the same Java gateway exception as above. I have also installed anaconda as I heard this may help unite python and java, again no success...</p>
  
  <p>Any suggestions about what to try next? I am at a loss.</p>