title: how to pass python package to spark job and invoke main file from package with arguments
body: <p>I have my python code with a structure like,</p>
  
  <pre><code>Project1
  --src
  ----util.py
  ----job1.py
  ----job2.py
  --config
  ----config1.json
  ----config2.json
  </code></pre>
  
  <p>I want to run this job1 in spark but these I just cannot invoke job1.py because its dependent on other files like util.py and job2.py and config files and thus I need to pass complete package as an input to spark. </p>
  
  <p>I tried running <code>spark-submit job1.py</code> but it fails with dependencies like job2.py and util.py because they are not available to executors. </p>
  
  <p>Based on spark documentation, I see --files is an option to do this but it works by passing all filenames to spark-submit which looks difficult if number of files in codebase in future. </p>
  
  <p>Another option I see is passing code zip file with --archive option but still it fails because not able to reference files in zip. </p>
  
  <p>So Can anyone suggest any other way to run such codebase in spark?</p>