Module ImportError using PySpark
Question: I have a pyspark job (spark 1.6.1, python 2.7). The basic structure is:

    
    
    spark_jobs/
      __init__.py
      luigi_workflow.py
      my_pyspark/
        __init__.py
        __main__.py
        spark_job.py
        stuff/
          __init__.py
          module1.py
          module2.py
        analytics/
          __init__.py
          analytics.py
    

In my `spark_job.py` I have:

    
    
    from dir1.module1 import func1
    from dir1.module2 import func2
    from analytics.analytics import some_analytics_func
    ...
    func1(...)
    func2(...)
    some_analytics_func(...)
    ...
    

When I launch the spark job, `func1` and `func2` execute perfectly, but then I
get:

`ImportError: No module named analytics.analytics`

This has been driving me absolutely insane. Any help would be appreciated.

Note: I'm launching with a wrapper around `spark-submit` and designating the
path with `python -m spark_jobs.my_pyspark`


Answer: I don't understand where `dir1` is coming from? Shouldn't it be `from
my_pyspark.stuff.module1 import func1`? Have you tried this before `from
my_pyspark.analytics.analytics import some_analytics_func`? Since you are
using Luigi, you can also try to build the package through
[setup.py](https://docs.python.org/2/install/).

Hope this helps! I had this problems before but it can be solved.

