Pants includes OS X specific Python wheels
Question: **TLDR** : Pants fetches OS X specific wheels bc I'm developing on Mac. How
can I avoid this, or specify that I will deploy to Ubuntu?

**Full story** :

Trying to package a Python application with Pants. Going great so far, but ran
into a problem which I've been stuck at for a while. I'm developing on a
macbook but deploying to EC2 Ubuntu.

Here's what I've done so far:

  1. Created virtualenv.
  2. Added BUILD files to applications, with the [suggested 3rd party pattern](https://pantsbuild.github.io/3rdparty_py.html) for third party packages.
  3. Ran `./pants run.py backend:admin_server` which runs fine and generated `dist/admin_server.pex`
  4. Scp that .pex onto a fresh EC2 Ubuntu box.

However when I run the application there, I get:

    
    
    Failed to execute PEX file, missing compatible dependencies for:
        mysql-python
        pycrypto
    

The problem seems to be that Pants takes OS X specific wheels for these 2:

pex: - MySQL_python-1.2.5-cp27-none-macosx_10_11_intel.whl pex: -
pycrypto-2.6.1-cp27-none-macosx_10_11_intel.whl

How can I avoid that, or specify which OS they should run on?

Here's the full output:

    
    
    ubuntu@ip-***:~$ export PEX_VERBOSE=1
    ubuntu@ip-***:~$ python admin_server.pex
    pex: Found site-library: /usr/local/lib/python2.7/dist-packages
    pex: Found site-library: /usr/lib/python2.7/dist-packages
    pex: Tainted path element: /usr/local/lib/python2.7/dist-packages
    pex: Tainted path element: /usr/lib/python2.7/dist-packages
    pex: Scrubbing from site-packages: /usr/local/lib/python2.7/dist-packages
    pex: Scrubbing from site-packages: /usr/lib/python2.7/dist-packages
    pex: Scrubbing from user site: /home/ubuntu/.local/lib/python2.7/site-packages
    pex: Failed to resolve a requirement: MySQL-python==1.2.5
    pex: Failed to resolve a requirement: pycrypto==2.6.1
    pex: Unresolved requirements:
    pex:   - mysql-python
    pex:   - pycrypto
    pex: Distributions contained within this pex:
    pex:   - six-1.10.0-py2.py3-none-any.whl
    pex:   - protobuf-2.6.1-py2.7.egg
    pex:   - setuptools-19.5-py2.py3-none-any.whl
    pex:   - MySQL_python-1.2.5-cp27-none-macosx_10_11_intel.whl
    pex:   - pycrypto-2.6.1-cp27-none-macosx_10_11_intel.whl
    pex:   - futures-3.0.4-py2-none-any.whl
    pex:   - webapp2-2.5.2-py2-none-any.whl
    pex:   - requests-2.9.0-py2.py3-none-any.whl
    pex:   - jmespath-0.9.0-py2.py3-none-any.whl
    pex:   - beautifulsoup4-4.4.1-py2-none-any.whl
    pex:   - python_dateutil-2.4.2-py2.py3-none-any.whl
    pex:   - boto3-1.2.3-py2.py3-none-any.whl
    pex:   - WebOb-1.5.1-py2.py3-none-any.whl
    pex:   - cssutils-1.0.1-py2-none-any.whl
    pex:   - webapp2_static-0.1-py2-none-any.whl
    pex:   - Paste-2.0.2-py2-none-any.whl
    pex:   - docutils-0.12-py2-none-any.whl
    pex:   - botocore-1.3.22-py2.py3-none-any.whl
    pex:   - protobuf_to_dict-0.1.0-py2-none-any.whl
    Failed to execute PEX file, missing compatible dependencies for:
    mysql-python
    pycrypto
    

PS: to make sure I didn't include my versions of the python libraries, I pip
uninstalled both PyCrypto and MySQL-Python.


Answer: One of the nice things about distributing your project as a PEX file is that
you can prepare it to run on multiple platforms. For example, one PEX can run
on both Linux and Mac platforms. For many projects, there is nothing special
to do other than build a PEX. But when your project has dependencies on
platform specific binary code, you will need to perform some extra steps.

One example of a library that contains platform specific code is the `psutil`
library. It contains C code that it compiled into a shared library when the
module is installed. To create a PEX file that contains such dependencies, you
must first provide a pre-built version of that library for all platforms other
than the one where you are running pants.

The easiest way to pre-build libraries is to use the pip tool to build wheels.

This recipe assumes the following:

  * You want to build a multi-platform pex to run on both Linux and mac You are going to pre-build the libraries in the Linux environment, then build the PEX on the mac environment. 
  * Your project directory lives under ~/src/cookbook

Let’s take a simple program that references a library and create a pex from
it.

    
    
    #  src/python/ps_example/main.py
    import psutil
    
    for proc in psutil.process_iter():
        try:
            pinfo = proc.as_dict(attrs=['pid', 'name'])
        except psutil.NoSuchProcess:
            pass
        else:
            print(pinfo)
    

With Pants, you can define an executable by defining a python_binary target in
a BUILD file:

    
    
    # src/python/ps_example/BUILD
    python_binary(name='ps_example',
      source = 'main.py',
      dependencies = [
        ':psutil',  # defined in requirements.txt
      ],
    )
    
    # Defines targets from specifications in requirements.txt
    python_requirements()
    

In the same directory, list the python libraries in a requirements.txt file:

    
    
    # src/python/ps_example/requirements.txt 
    psutil==3.1.1
    

Now, to to make the multi-platform pex, you'll need access to a Linux box to
create the linux version of psutil wheel. Copy the requirements.txt file to
the linux machine, then, execute the pip tool:

    
    
    linux $ mkdir ~/src/cookbook/wheelhouse
    linux $ pip wheel -r src/python/multi-platform/requirements.txt  \
        --wheel-dir=~/src/cookbook/wheelhouse
    

This will create a platform specific wheel file.

    
    
    linux $ ls ~/src/cookbook/wheelhouse/
    psutil-3.1.1-cp27-none-linux_x86_64.whl
    

Now, you will need to copy the platform specific wheel over to the machine
where you want to build your multi-platform pex (in this case, your mac
laptop). If you use this recipe on a regular basis, you will probably want to
configure a Python Respository to store your pre-built libraries.

We’ll use the same BUILD file setup as in above, but modify python_binary to
specify the `platforms=` parameter.

    
    
    # src/python/ps_example/BUILD
    python_binary(name='ps_example',
      source = 'main.py',
      dependencies = [
        ':psutil',  # defined in requirements.txt
      ],
      platforms=[
        'linux-x86_64',
        'macosx-10.7-x86_64',
      ],
    )
    
    # Defines targets from specifications in requirements.txt
    python_requirements()
    

You will also need to tell pants where to find the pre-built python packages.
Edit `pants.ini` and add:

    
    
    [python-repos]
    repos: [
        "%(buildroot)s/wheelhouse/"
      ]
    

Now, copy the file `psutil-3.1.1-cp27-none-linux_x86_64.whl` over to the mac
workstation and place it in a directory named `wheelhouse/` under the root of
your repo.

Once this is done you can now build the multi-platform pex with

    
    
    mac $ ./pants binary src/python/py_example
    

You can verify that libraries for both mac and Linux are included in the pex
by unzipping it:

    
    
    mac $ unzip -l dist/ps_example.pex | grep psutil
        17290  12-21-15 22:09   .deps/psutil-3.1.1-cp27-none-linux_x86_64.whl/psutil-3.1.1.dist-info/DESCRIPTION.rst
        19671  12-21-15 22:09   .deps/psutil-3.1.1-cp27-none-linux_x86_64.whl/psutil-3.1.1.dist-info/METADATA
         1340  12-21-15 22:09   .deps/psutil-3.1.1-cp27-none-linux_x86_64.whl/psutil-3.1.1.dist-info/RECORD
          103  12-21-15 22:09  
    ...   .deps/psutil-3.1.1-cp27-none-macosx_10_11_intel.whl/psutil-3.1.1.dist-info/DESCRIPTION.rst
        19671  12-21-15 22:09   .deps/psutil-3.1.1-cp27-none-macosx_10_11_intel.whl/psutil-3.1.1.dist-info/METADATA
         1338  12-21-15 22:09   .deps/psutil-3.1.1-cp27-none-macosx_10_11_intel.whl/psutil-3.1.1.dist-info/RECORD
          109  12-21-15 22:09   
    ...
    

