title: Apache Beam local Python dependencies
body: <p>I have local Python package which I want to use inside of the Apache Beam pipeline with DataFlow Runner.
  I tried to follow instructions provided in the documentation: <a href="https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/" rel="nofollow noreferrer">https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/</a> (section <em>Local or non-PyPI Dependencies</em>) but without success.</p>
  
  <p>My package has the following structure:</p>
  
  <pre><code>my_common
  ├── __init__.py
  └── shared
      ├── __init__.py
      └── something.py
  </code></pre>
  
  <p>With <code>something.py</code> file containing:</p>
  
  <pre><code>def hello_world():
      return "Hello"
  </code></pre>
  
  <p>Package was build using <code>python setup.py sdist</code> command.</p>
  
  <p>Now, I have Apache Beam pipeline configured as follows:</p>
  
  <pre><code>pipeline_parameters = [
      '--project', project_id,
      '--staging_location', staging_location,
      '--temp_location', temp_location,
      '--max_num_workers', 1,
      "--extra_package", "/absolute/path/to/my/package/my_common-1.0.tar.gz"
  ]
  
  
  p = beam.Pipeline("DataFlowRunner", argv=pipeline_parameters)
  # rest of the pipeline definition
  </code></pre>
  
  <p>One of the pipeline map functions has the following code, which uses my module:</p>
  
  <pre><code>from my_common.shared import something
  logging.info(something.hello_world())
  </code></pre>
  
  <p>Every time when I schedule this pipeline to DataFlow, I get the following error:</p>
  
  <pre><code>ImportError: No module named shared
  </code></pre>
  
  <p>Interesting part is that when I install this package (from .tar.gz) file on another environment, I can import and run function from it without any problems. It seems to me that DataFlow does not install the package before running the pipeline. </p>
  
  <p><strong>What is the proper way of managing and deploying local Python dependencies to Google DataFlow?</strong></p>
  
  <p><em>//update:</em>
  The solution described in <a href="https://stackoverflow.com/a/46605344/1955346">https://stackoverflow.com/a/46605344/1955346</a> is not sufficient for my use case, since I need to have my local packages in completely different folder and <code>setup.py</code> for my pipeline already has some contents (I cannot use setup.py of external package as suggested there).</p>