How do I set the driver's python version in spark?
I just faced the same issue and these are the steps that I follow in order to provide Python version. I wanted to run my PySpark jobs with Python 2.7 instead of 2.6. 

Go to the folder where $SPARK_HOME is pointing to (in my case is /home/cloudera/spark-2.1.0-bin-hadoop2.7/)
Under folder conf, there is a file called spark-env.sh. In case you have a file called spark-env.sh.template you will need to copy the file to a new file called spark-env.sh.
Edit the file and write the next three lines

export PYSPARK_PYTHON=/usr/local/bin/python2.7
export PYSPARK_DRIVER_PYTHON=/usr/local/bin/python2.7
export SPARK_YARN_USER_ENV="PYSPARK_PYTHON=/usr/local/bin/python2.7"

Save it and launch your application again :) 

In that way, if you download a new Spark standalone version, you can set the Python version which you want to run PySpark to.
I'm using spark 1.4.0-rc2 so I can use python 3 with spark.  If I add export PYSPARK_PYTHON=python3 to my .bashrc file, I can run spark interactively with python 3.  However, if I want to run a standalone program in local mode, I get an error:

Exception: Python in worker has different version 3.4 than that in
driver 2.7, PySpark cannot run with different minor versions

How can I specify the version of python for the driver?  Setting export PYSPARK_DRIVER_PYTHON=python3 didn't work.