Adding h5 files in a zip to use with PySpark
Found that you can add the additional files to all the workers by using --files argument in spark-submit.
I am using PySpark 1.6.1 for my spark application. I have additional modules which I am loading using the argument --py-files. I also have a  h5 file which I need to access from one of the modules for initializing the ApolloNet.
Is there any way I could access those files from the modules if I put them in the same archive? I tried this approach but it was throwing an error because the files are not there in every worker. I can think of copying the file to each of the workers but I want to know if there are better ways to do it?