How to deal with Linux/Python dependencies?
Question: Due to lack of support for some libraries I want to use, I moved some Python
development from Windows to Linux development. I've spent most of the day
messing about getting nowhere with dependencies.

**The question**

Whenever I pick up Linux, I usually run into some kind of dependency issue,
usually with development libraries, whether they're installed via apt-get,
easy_install or pip. I can waste days on what should be simple tasks, spending
longer on getting libraries to work than writing code. **Where can I learn
about strategy for dealing with these kind of issues rather than aimlessly
googling for someone who's come across the same problem before?**

* * *

**An example**

Just one example: I wanted to generate some QR codes. So, I thought I'd use
[github.com/bitly/pyqrencode](https://github.com/bitly/pyqrencode) which is
based on [pyqrcode.sourceforge.net](http://pyqrcode.sourceforge.net/) but
supposedly without the Java dependencies. There are others
([pyqrnative](http://code.google.com/p/pyqrnative/),
[github.com/Arachnid/pyqrencode](https://github.com/Arachnid/pyqrencode/)) but
that one seemed like the best bet for my needs.

So, I found the package on [pypi](http://pypi.python.org/pypi/pyqrencode) and
thought using that would make life easier:

(I've perhaps made life more difficult for myself by using virtualenv to keep
things neat and tidy.)

    
    
    (myenv3)mat@ubuntu:~/myenv3$ bin/pip install pyqrencode
    Downloading/unpacking pyqrencode
      Downloading pyqrencode-0.2.tar.gz
      Running setup.py egg_info for package pyqrencode
    
    Installing collected packages: pyqrencode
      Running setup.py install for pyqrencode
        building 'qrencode' extension
        gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c qrencode.c -o build/temp.linux-i686-2.7/qrencode.o
        gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions build/temp.linux-i686-2.7/qrencode.o -lqrencode -o build/lib.linux-i686-2.7/qrencode.so
    
    Successfully installed pyqrencode
    Cleaning up...
    

(I guess I probably `sudo apt-get install libqrencode-dev` at some point prior
to that too.)

So then I tried to run the test script:

    
    
    (myenv3)mat@ubuntu:~/myenv3$ python test_qr.py 
    Traceback (most recent call last):
      File "test_qr.py", line 1, in <module>
        from qrencode import Encoder
      File "qrencode.pyx", line 1, in init qrencode (qrencode.c:1520)
    ImportError: No module named ImageOps
    

:(

Well, [investigations](http://www.google.com/search?q=ImageOps) revealed that
ImageOps appears to be part of PIL...

    
    
    (myenv3)mat@ubuntu:~/myenv3$ pip install pil
    Downloading/unpacking pil
      Downloading PIL-1.1.7.tar.gz (506Kb): 122Kb downloaded
    Operation cancelled by user
    Storing complete log in /home/mat/.pip/pip.log
    (myenv3)mat@ubuntu:~/myenv3$ bin/pip install pil
    Downloading/unpacking pil
      Downloading PIL-1.1.7.tar.gz (506Kb): 506Kb downloaded
      Running setup.py egg_info for package pil
        WARNING: '' not a valid package name; please use only.-separated package names in setup.py
    
    Installing collected packages: pil
      Running setup.py install for pil
        WARNING: '' not a valid package name; please use only.-separated package names in setup.py
        building '_imaging' extension
        gcc ...
        building '_imagingmath' extension
        gcc ...
        --------------------------------------------------------------------
        PIL 1.1.7 SETUP SUMMARY
        --------------------------------------------------------------------
        version       1.1.7
        platform      linux2 2.7.1+ (r271:86832, Apr 11 2011, 18:05:24)
                      [GCC 4.5.2]
        --------------------------------------------------------------------
        *** TKINTER support not available
        *** JPEG support not available
        *** ZLIB (PNG/ZIP) support not available
        *** FREETYPE2 support not available
        *** LITTLECMS support not available
        --------------------------------------------------------------------
        To add a missing option, make sure you have the required
        library, and set the corresponding ROOT variable in the
        setup.py script.
    
        To check the build, run the selftest.py script.
        ...
    Successfully installed pil
    Cleaning up...
    

Hmm, PIL's installed but hasn't picked up the libraries I installed with `sudo
apt-get install libjpeg62 libjpeg62-dev libpng12-dev zlib1g zlib1g-dev`
earlier. I'm not sure how to tell pip to feed the library locations to
setup.py. Googling suggests a variety of
[ideas](http://www.cleverkoala.com/2011/09/how-to-install-python-imaging-in-a-
virtualenv-with-no-site-packages/) which I've tried, but none of them seem to
help much other than to send me round in circles.

[Ubuntu 11.04: Installing PIL into a virtualenv with
PIP](http://stackoverflow.com/questions/6138848/ubuntu-11-04-installing-pil-
into-a-virtualenv-with-pip) suggests using the
[pillow](http://pypi.python.org/pypi/Pillow) package instead, so let's try
that:

    
    
    (myenv3)mat@ubuntu:~/myenv3$ pip install pillow
    Downloading/unpacking pillow
      Downloading Pillow-1.7.5.zip (637Kb): 637Kb downloaded
      Running setup.py egg_info for package pillow
    
        ...
    Installing collected packages: pillow
      Running setup.py install for pillow
        building '_imaging' extension
        gcc ...
        --------------------------------------------------------------------
        SETUP SUMMARY (Pillow 1.7.5 / PIL 1.1.7)
        --------------------------------------------------------------------
        version       1.7.5
        platform      linux2 2.7.1+ (r271:86832, Apr 11 2011, 18:05:24)
                      [GCC 4.5.2]
        --------------------------------------------------------------------
        *** TKINTER support not available
        --- JPEG support available
        --- ZLIB (PNG/ZIP) support available
        --- FREETYPE2 support available
        *** LITTLECMS support not available
        --------------------------------------------------------------------
        To add a missing option, make sure you have the required
        library, and set the corresponding ROOT variable in the
        setup.py script.
    
        To check the build, run the selftest.py script.
        ...
    Successfully installed pillow
    Cleaning up...
    

Well, we seem to have the JPEG and PNG support this time, yay!

    
    
    (myenv3)mat@ubuntu:~/myenv3$ python test_qr.py 
    Traceback (most recent call last):
      File "test_qr.py", line 1, in <module>
        from qrencode import Encoder
      File "qrencode.pyx", line 1, in init qrencode (qrencode.c:1520)
    ImportError: No module named ImageOps
    

Still no ImageOps though. Now I'm stumped, is ImageOps missing from pillow, or
is it a different problem that was also there with pil.


Answer: I see two separate problems here:

  1. Keeping track of all the python modules you need for your project.

  2. Keeping track of all the dynamic libraries you need for the python modules in your project.

For the first problem, I have found that
[buildout](http://www.buildout.org/%20buildout) is good help, althought it
takes a litle while to grasp.

In your case, I would start by creating a directory for my new project. I
would then go into that directory and download _bootstrap.py_

    
    
    wget http://python-distribute.org/bootstrap.py 
    

I would then create a _buildout.cfg_ file:

    
    
    [buildout]
    parts = qrproject
            python
    eggs = pyqrencode
    
    [qrproject]
    recipe = z3c.recipe.scripts
    eggs = ${buildout:eggs}
    entry-points= qrproject=qrprojectmodule:run
    extra-paths = ${buildout:directory}
    
    # This is a simple way of creating an interpreter that will have
    # access to all the eggs / modules that this project uses.
    [python]
    recipe = z3c.recipe.scripts
    interpreter = python
    eggs = ${buildout:eggs}
    extra-paths = ${buildout:directory}
    

In this _buildout.cfg_ I'm referencing the module _qrprojectmodule_ (in
_entry-points_ under _[qrproject]_. This will create a bin/qrproject that runs
the function _run_ in the module _qrprojectmodule_. So I will also create the
file _qrprojectmodule.py_

    
    
    import qrencode
    
    def run():
        print "Entry point for qrproject. Happily imports qrencode module"
    

Now it's time to run _bootstrap.py_ with the python binary you want to use:

    
    
    python bootstrap.py
    

Then run the generated _bin/buildout_

    
    
    bin/buildout
    

This will create two additional binaries in the _bin/_ directory -
_bin/qrproject_ and _bin/python_. The former is your project's main binary. It
will be created automatically each time you run buildout and will have all the
modules and eggs you want loaded. The second is simply a convenient way to get
a python prompt where all your modules and eggs are loaded, for easy
debugging. The fine thing here is that bin/buildout will automatically install
any python eggs that the eggs (in your case pyqrencode) have specified as
dependencies.

Actually, you will probably get a compilation error in the step where you run
_bin/buildout_. This is because you need to address problem 2: All dynamic
libraries being available on your system. On Linux, it's usually best to get
help from your packaging system. I'm going to assume you're using a Debian
derivate such as Ubuntu here.

The pyqrencode web site specifies that you need the libqrencode library for
pyqrencode to work. So I used my package manager to search for that:

    
    
    $ apt-cache search libqrencode
    libqrencode-dev - QR Code encoding library -- development
    libqrencode3 - QR Code encoding library
    qrencode - QR Code encoder into PNG image
    

In this case, I want the -dev package, as that installs linkable libraries and
header files required to compile python C-modules. Also, the dependency system
in the package manager will make sure that if I install _libqrencode-dev_ , I
will also get _libqrencode3_ , as that is required at runtime, i.e. after
compilation of the module.

So, I install the package:

    
    
    sudo apt-get install libqrencode-dev
    

Once that has completed, rerun bin/buildout and the pyqrencode module will
(hopefully) compile and install successfully. Now try to run _bin/qrproject_

    
    
    $ bin/qrproject 
    Entry point for qrproject. Happily imports qrencode module
    

Success! :-)

So, in summary:

  1. Use buildout to automatically download and install all the python modules/eggs you need for your project.

  2. Use your system's package manager to install any dynamic (C) libraries required by the python modules you use.

Be aware that in many cases there are already packaged versions of your python
modules available in the package system. For example, pil is available by
installing the _python-imaging_ package on Ubuntu. In this case, you don't
need to install it via buildout, and you don't need to worry about libraries
being available - the package manager will install all dependencies required
for the module to run. Doing it via buildout can however make it easier to
distribute your project and make it run on other systems.

