Spark + Python - how to set the system environment variables?
Question: I'm on spark-1.4.1. How can I set the system environment variables for Python?

For instance, in R,

    
    
    Sys.setenv(SPARK_HOME = "C:/Apache/spark-1.4.1")
    .libPaths(c(file.path(Sys.getenv("SPARK_HOME"), "R", "lib"), .libPaths()))
    

What about in Python?

    
    
    import os
    import sys
    
    from pyspark.sql import SQLContext
    
    sc = SparkContext(appName="PythonSQL")
    sqlContext = SQLContext(sc)
    
    # Set the system environment variables.
    # ref: https://github.com/apache/spark/blob/master/examples/src/main/python/sql.py
    if len(sys.argv) < 2:
        path = "file://" + \
            os.path.join(os.environ['SPARK_HOME'], "examples/src/main/resources/people.json")
    else:
        path = sys.argv[1]
    
    # Create the DataFrame
    df = sqlContext.jsonFile(path)
    
    # Show the content of the DataFrame
    df.show()
    

I get this error,

df is not defined.

[![enter image description
here](http://i.stack.imgur.com/Aduko.png)](http://i.stack.imgur.com/Aduko.png)

Any ideas?


Answer: Just try it like this: <https://spark.apache.org/docs/latest/sql-programming-
guide.html#creating-dataframes>

By providing `path = "examples/src/main/resources/people.json"` as parameter
to `df = sqlContext.jsonFile(path)`

If you don't provide arguments when you run your python script, then it will
go into `if len(sys.argv) < 2:`, this requires you to have defined
`SPARK_HOME` as a system variable. If not, it won't find your specified .json
file. Which seems to be your problem.

