spark-submit python file ‘home/.python-eggs’ permission denied
Question: I had a problem when I use spark-submit to run a python file.When the 'map'
code run in 'executor', the problem like this :

    
    
    Traceback (most recent call last):
    File "/usr/lib64/python2.7/runpy.py", line 151, in _run_module_as_main
      mod_name, loader, code, fname = _get_module_details(mod_name)
    File "/usr/lib64/python2.7/runpy.py", line 101, in _get_module_details
      loader = get_loader(mod_name)
    File "/usr/lib64/python2.7/pkgutil.py", line 464, in get_loader
      return find_loader(fullname)
    File "/usr/lib64/python2.7/pkgutil.py", line 474, in find_loader
      for importer in iter_importers(fullname):
    File "/usr/lib64/python2.7/pkgutil.py", line 430, in iter_importers
      __import__(pkg)
    File "/data8/yarn/local-dir/usercache/bo.feng/appcache/application_1448854352032_70810/container_1448854352032_70810_01_000002/pyspark.zip/pyspark/__init__.py", line 41, in <module>
    File "/data8/yarn/local-dir/usercache/bo.feng/appcache/application_1448854352032_70810/container_1448854352032_70810_01_000002/pyspark.zip/pyspark/context.py", line 35, in <module>
    File "/data8/yarn/local-dir/usercache/bo.feng/appcache/application_1448854352032_70810/container_1448854352032_70810_01_000002/pyspark.zip/pyspark/rdd.py", line 51, in <module>
    File "/data8/yarn/local-dir/usercache/bo.feng/appcache/application_1448854352032_70810/container_1448854352032_70810_01_000002/pyspark.zip/pyspark/shuffle.py", line 33, in <module>
    File "build/bdist.linux-x86_64/egg/psutil/__init__.py", line 89, in <module>
    File "build/bdist.linux-x86_64/egg/psutil/_pslinux.py", line 24, in <module>
    File "build/bdist.linux-x86_64/egg/_psutil_linux.py", line 7, in <module>
    File "build/bdist.linux-x86_64/egg/_psutil_linux.py", line 4, in __bootstrap__
    File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 945, in resource_filename
      self, resource_name
    File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 1633, in get_resource_filename
      self._extract_resource(manager, self._eager_to_zip(name))
    File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 1661, in _extract_resource
      self.egg_name, self._parts(zip_path)
    File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 1025, in get_cache_path
      self.extraction_error()
    File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 991,     inextraction_error
      raise err
      pkg_resources.ExtractionError: Can't extract file(s) to egg cache
      The following error occurred while trying to extract file(s) to the Python egg
      cache:
        [Errno 13] Permission denied: '/home/.python-eggs' 
      The Python egg cache directory is currently set to:  
       /home/.python-eggs  
     Perhaps your account does not have write access to this directory?  You can
     change the cache directory by setting the PYTHON_EGG_CACHE environment
     variable to point to an accessible directory.
    

I set the PYTHON_EGG_CACHE environment variable to every executor,and I also
write 'os.environ['PYTHON_EGG_CACHE'] = "/tmp/"' in my program,but the problem
is still happen.

My code :

    
    
    import os,sys
    print "env::::"+os.environ['PYTHON_EGG_CACHE']
    from pyspark import SparkConf, SparkContext
    # Load and parse the data
    def parsePoint(line):
        import os
        print "env::::"+os.environ['PYTHON_EGG_CACHE']
        os.environ['PYTHON_EGG_CACHE'] = "/tmp/"
        values = [float(x) for x in line.split(' ')]
        return line
    
    if __name__ == "__main__":
        os.environ['PYTHON_EGG_CACHE'] = "/tmp/"
        print "env::::"+os.environ['PYTHON_EGG_CACHE']
        conf = SparkConf()
        sc = SparkContext(conf = conf)
        data = sc.textFile(sys.argv[1])
        parsedData = data.map(parsePoint)
        parsedData.collect()
    

I run this python program in 'standalone' model and succeeded. This is my
submit command:

    
    
    spark-submit --name test_py --master yarn-client testpy.py input/sample_svm_data.txt
    

Is the yarn's problem?


Answer: I solved this problem： unzip the pyspark.zip then find rdd.py file open this
file , under "import os" line ,add code as :

    
    
    os.environ['PYTHON_EGG_CACHE'] = '/tmp/.python-eggs/'
    os.environ['PYTHON_EGG_DIR']='/tmp/.python-eggs/'
    

save file and zip pyspark

