how to control the import paths in Python?
Question: I am new to Python. I am at a company where they built a large system in
Python. They use a proprietary system to manage the paths when the system is
running, but now I have been asked to build a standalone script that interacts
with some of the code in their system. Sadly, my standalone script won't be
running under the path-manager they use, so I need to figure out the paths on
my own.

So, for instance, I have this line:

    
    
    from hark.tasks import REPLY_LINE
    

This is actually copied from some of their older code. In this case, the
script can find hark, but hark has an `__init__.py` file, and that is where
the problems start. So I get this:

    
    
    meg/src/python2/hark/hark/__init__.py in <module>()
          5 from flask import jsonify, render_template, request
          6 import jinja2
    ----> 7 import logbook, logbook.compat
          8 
          9 from healthhark.context import Ghost, g
    

The project that they built actually includes logbook 3 times. If I do:

    
    
    find . -name "*logbook*"
    

I see:

    
    
    meg/zurge/opt/python2.7/lib/python2.7/site-packages/logbook
    meg/zurge/opt/python2.7-hark/lib/python2.7/site-packages/logbook
    meg/zurge/opt/python3.4/lib/python3.4/site-packages/logbook
    

Like I said, they have a proprietary path manager that usually tells each
piece of code where it can find the packages that it should include, but I am
building a standalone app.

I don't know much about Python, but I am wondering if their is an idiomatic
and Pythonic way of including packages that are in such distant directories?

And, before anyone suggests `pip install`, we don't rely on global installs at
all.


Answer: The best solution would probably be
[virtualenv](https://pypi.python.org/pypi/virtualenv) or
[virtualenvwrapper](https://pypi.python.org/pypi/virtualenvwrapper). This
would allow you to define an environment which contains all of the libraries
that your script requires. This would not be global.

This can be done as follows:

  * Create a [requirements.txt file](http://pip.readthedocs.org/en/latest/user_guide.html#requirements-files) defining the required libraries
  * Install pip and virtualenvwrapper
  * `source /usr/local/bin/virtualenvwrapper.sh`
  * `workon hark_task_script_env || ( mkvirtualenv hark_task_script_env && pip install -r requirements.txt )`
  * `python your-script.py`

The virtualenvwrapper installs the environment into a folder in your home
directory. The plain virtualenv library installs the environment into a folder
in the project. Other than that they are equivalent.

I would really recommend using them over a proprietry package manager. If you
must use the proprietry package manager then it is reasonable to have access
to the package loader for it!

If this really isn't satisfactory then you can hack a package loader as
follows (assuming you use a *nix system). This is a shell script written in
zsh. Hopefully it is clear enough for you to rewrite it in a supported shell
if your system does not have that available:

    
    
    #!/bin/zsh
    
    setopt extended_glob
    
    function find_package_init_files () {
        locate __init__.py
    }
    
    # Get the containing folder of a file or a folder
    function file_or_folder_to_parent_folder () {
        while read file_or_folder
        do
            echo ${file_or_folder:h}
        done
    }
    
    # Exclude folders where the parent folder also has an __init__.py file in it
    function exclude_inner_packages () {
        while read init_file_folder
        do
            init_file_parent_folder=${init_file_folder:h}
            if [ ! -e ${init_file_parent_folder}/__init__.py ]
            then
                echo ${init_file_folder}
            fi
        done
    }
    
    # This produces an array of all folders
    # that contain at least one python package
    function get_distinct_python_package_folders () {
        find_package_init_files |
            file_or_folder_to_parent_folder |
            exclude_inner_packages |
            file_or_folder_to_parent_folder |
            sort |
            uniq
    }
    
    PYTHONPATH=${(j/:/)$(get_distinct_python_package_folders)} YOUR_SCRIPT_HERE
    

You may well need to update this script to put the default python path first,
and remember that this is an incredibly clumsy approach. If there are multiple
versions of libraries installed on a system then the one that you may end up
using will be ill defined.

Using a proper package manager, even if that is the proprietry one, is the
best way.

