title: spark.executorEnv doesn't seem to take any effect
body: <p>According to the Spark <a href="http://spark.apache.org/docs/latest/configuration.html" rel="nofollow">docs</a>, there is a way to pass environment variables to spawned executors:</p>
  
  <blockquote>
    <p>spark.executorEnv.[EnvironmentVariableName] Add the environment variable specified by EnvironmentVariableName to the Executor process. The user can specify multiple of these to set multiple environment variables.</p>
  </blockquote>
  
  <p>I'm trying to direct my pyspark app to use a specific python executable(anaconda environment with numpy etc etc), which is usually done by altering the <code>PYSPARK_PYTHON</code> variable in <code>spark-env.sh</code>. Although this way works, shipping new config to all the cluster nodes  every time I want to switch a virtualenv looks like a huge overkill.</p>
  
  <p>That's why I tried to pass PYSPARK_PYTHON in the following way:</p>
  
  <pre class="lang-sh prettyprint-override"><code>uu@e1:~$ PYSPARK_DRIVER_PYTHON=ipython pyspark --conf \
  spark.executorEnv.PYSPARK_PYTHON="/usr/share/anaconda/bin/python" \
  --master spark://e1.local:7077
  </code></pre>
  
  <p>But it doesn't seem to work:</p>
  
  <pre class="lang-python prettyprint-override"><code>In [1]: sc._conf.getAll()
  Out[1]: 
  [(u'spark.executorEnv.PYSPARK_PYTHON', u'/usr/share/anaconda/bin/python'),
   (u'spark.rdd.compress', u'True'),
   (u'spark.serializer.objectStreamReset', u'100'),
   (u'spark.master', u'spark://e1.local:7077'),
   (u'spark.submit.deployMode', u'client'),
   (u'spark.app.name', u'PySparkShell')]
  
  In [2]: def dummy(x):
      import sys
      return sys.executable
     ...: 
  
  In [3]: sc.parallelize(xrange(100),50).map(dummy).take(10)
  
  Out[3]: 
  ['/usr/bin/python2.7',
   '/usr/bin/python2.7',
   '/usr/bin/python2.7',
   '/usr/bin/python2.7',
   '/usr/bin/python2.7',
   '/usr/bin/python2.7',
   '/usr/bin/python2.7',
   '/usr/bin/python2.7',
   '/usr/bin/python2.7',
   '/usr/bin/python2.7']
  </code></pre>
  
  <p>My <code>spark-env.sh</code> does not have <code>PYSPARK_PYTHON</code> configured, so this is just the default python that gets called. Some additional info: it's spark 1.6.0 standalone mode cluster.</p>
  
  <p>Am I missing something important here?</p>