python-project-packaging-with-pandas
#### python-project-packaging-with-pandas


#### Problem description
I have created a Spark project in Python in which I am using the module pandas and packaged the spark project using wheel as per the these steps: -
>pip wheel -r requirements.txt -w dist
it will create 'wheel' files for all explicit and implicit dependencies from our 'requirement.txt' file and put them into 'dist' directory.
To build our project artifacts.
>python setup.py bdist_wheel
This will create another 'wheel' (zip file underneath) in 'dist' directory with our application files. Renamed our 'wheel' files as zip files and provide them to '--py-files' option to submit the spark job.
This is spark submit command I am submitting to execute my job.
>spark-submit --py-files certifi-2018.11.29-py2.py3-none-any.zip,findspark-1.3.0-py2.py3-none-any.zip,idna-2.8-py2.py3-none-any.zip,kafka-1.3.5-py2.py3-none-any.zip,Logr-0.2.2-cp27-none-any.zip,mysql_connector-2.1.6-cp27-cp27mu-linux_x86_64.zip,panda-0.3.1-cp27-none-any.zip,py4j-0.10.7-py2.py3-none-any.zip,python_dateutil-2.7.5-py2.py3-none-any.zip,pytz-2018.7-py2.py3-none-any.zip,req-1.0.0-py2.py3-none-any.zip,requests-2.21.0-py2.py3-none-any.zip,setuptools-40.6.3-py2.py3-none-any.zip,six-1.12.0-py2.py3-none-any.zip,urllib3-1.24.1-py2.py3-none-any.zip,zope.interface-4.6.0-cp27-cp27mu-manylinux1_x86_64.zip,configparser-3.5.0-py2-none-any.zip,pandas-0.23.4-cp27-cp27mu-manylinux1_x86_64.zip,numpy-1.15.4-cp27-cp27mu-manylinux1_x86_64.zip,test_spark_submit-0.1.zip  driver.py test_spark_submit.subfolder.spark_script parameter_to_pass_to_module
Issue is when we fire this command it is failing with error while all other module dependencies are working fine (like kafka, findspark and etc.). 
It seems pandas is dependent on numpy and even numpy(numpy-1.15.4-cp27-cp27mu-manylinux1_x86_64.zip) is given in command but still it is not referring this.

> File "/home/harry/pandas-0.23.4-cp27-cp27mu-manylinux1_x86_64.zip/pandas/__init__.py", line 19, in <module>
ImportError: Missing required dependencies ['numpy']

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/harry/DataIngestionTool_dist/driver.py", line 44, in <module>
    raise Exception("Exception::Job %s failed with msg %s" %(args.job_name, str(e)))
Exception: Exception::Job dataIngestionTool.dataPrepartion.dataIngestion failed with msg Missing required dependencies ['numpy']
 
####
* I want to package in this way as I can manage all the dependencies effectively.
Please help me to resolve this issue.

#### Could you please check on pandas/__init__.py", line 19,  why module is null.