Cython and fortran - how to compile together without f2py
Question: **FINAL UPDATE**

This question is about how to write a `setup.py` that will compile a cython
module which accesses FORTRAN code directly, like C would. It was a rather
long and arduous journey to the solution, but the full mess is included below
for context.

**ORIGINAL QUESTION**

I have an extension which is a Cython file, which sets up some heap memory and
passes it to the fortran code, and a fortran file, which is a venerable old
module that I'd like to avoid reimplementing if I can.

The `.pyx` file compiles fine to C, but the cython compiler chokes on the
`.f90` file with the following error:

    
    
    $ python setup.py build_ext --inplace
    running build_ext
    cythoning delaunay/__init__.pyx to delaunay/__init__.c
    building 'delaunay' extension
    error: unknown file type '.f90' (from 'delaunay/stripack.f90')
    

Here's (the top half of) my setup file:

    
    
    from distutils.core import setup, Extension
    from Cython.Distutils import build_ext
    
    ext_modules = [
      Extension("delaunay",
        sources=["delaunay/__init__.pyx",
                 "delaunay/stripack.f90"])
    ]
    
    setup(
      cmdclass = {'build_ext': build_ext},
      ext_modules = ext_modules,
      ...
    )
    

_NOTE: I originally had the fortran file's location incorrectly specified
(without the directory prefix) but this breaks in exactly the same way after I
fixed that._

**Things I have tried:**

I found [this](http://stackoverflow.com/questions/6523767/how-do-i-get-setup-
py-test-to-use-a-specific-fortran-compiler), and tried passing in the name of
the fortran compiler (i.e. gfortran) like this:

    
    
    $ python setup.py config --fcompiler=gfortran build_ext --inplace
    usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
       or: setup.py --help [cmd1 cmd2 ...]
       or: setup.py --help-commands
       or: setup.py cmd --help
    
    error: option --fcompiler not recognized
    

And I've also tried removing `--inplace`, in case that was the problem (it
wasn't, same as the top error message).

So, how do I compile this fortran? Can I hack it into a `.o` myself and get
away with linking it? Or [is this a bug in
Cython](https://groups.google.com/forum/?fromgroups=#!topic/cython-
users/DP_FksDi6iM), which will force me to reimplement distutils or hack
around with the preprocessor?

**UPDATE**

So, having checked out the `numpy.distutils` packages, I understand the
problem a bit more. It seems that you have to

  1. Use cython to convert the .pyx files to cpython .c files,
  2. Then use an `Extension`/`setup()` combination that supports fortran, like `numpy`'s.

Having tried this, my `setup.py` now looks like this:

    
    
    from numpy.distutils.core import setup
    from Cython.Build import cythonize
    from numpy.distutils.extension import Extension
    
    cy_modules = cythonize('delaunay/sphere.pyx')
    e = cy_modules[0]
    
    ext_modules = [
      Extension("delaunay.sphere",
          sources=e.sources + ['delaunay/stripack.f90'])
    ]
    
    setup(
      ext_modules = ext_modules,
      name="delaunay",
      ...
    )
    

(note that I've also restructured the module a bit, since seemingly an
`__init__.pyx` is disallowed...)

Now is where things become buggy and platform-dependent. I have two testing
systems available - one Mac OS X 10.6 (Snow Leopard), using Macports Python
2.7, and one Mac OS X 10.7 (Lion) using the system python 2.7.

On Snow Leopard, the following applies:

This means that the module compiles (hurray!) (although there's no `--inplace`
for numpy, it seems, so I had to system-wide install the testing module :/)
but I still get a crash on `import` as follows:

    
    
      >>> import delaunay
      Traceback (most recent call last):
        File "<input>", line 1, in <module>
        File "<snip>site-packages/delaunay/__init__.py", line 1, in <module>
          from sphere import delaunay_mesh
      ImportError: dlopen(<snip>site-packages/delaunay/sphere.so, 2): no suitable image found.  Did find:
        <snip>site-packages/delaunay/sphere.so: mach-o, but wrong architecture
    

and on Lion, I get a compile error, following a rather confusing looking
compile line:

    
    
    gfortran:f77: build/src.macosx-10.7-intel-2.7/delaunay/sphere-f2pywrappers.f
    /usr/local/bin/gfortran -Wall -arch i686 -arch x86_64 -Wall -undefined dynamic_lookup -bundle build/temp.macosx-10.7-intel-2.7/delaunay/sphere.o build/temp.macosx-10.7-intel-2.7/build/src.macosx-10.7-intel-2.7/delaunay/spheremodule.o build/temp.macosx-10.7-intel-2.7/build/src.macosx-10.7-intel-2.7/fortranobject.o build/temp.macosx-10.7-intel-2.7/delaunay/stripack.o build/temp.macosx-10.7-intel-2.7/build/src.macosx-10.7-intel-2.7/delaunay/sphere-f2pywrappers.o -lgfortran -o build/lib.macosx-10.7-intel-2.7/delaunay/sphere.so
    ld: duplicate symbol _initsphere in build/temp.macosx-10.7-intel-2.7/build/src.macosx-10.7-intel-2.7/delaunay/spheremodule.o ldand :build /temp.macosx-10.7-intelduplicate- 2.7symbol/ delaunay/sphere.o _initsphere in forbuild architecture /i386
    temp.macosx-10.7-intel-2.7/build/src.macosx-10.7-intel-2.7/delaunay/spheremodule.o and build/temp.macosx-10.7-intel-2.7/delaunay/sphere.o for architecture x86_64
    

Now let's just step back a moment before we pore over the details here.
Firstly, I know there are a bunch of headaches over architecture clashes in
64-bit Mac OS X; I had to work very hard to get Macports Python working on the
Snow Leopard machine (just to upgrade from system python 2.6). I also know
that when you see `gfortran -arch i686 -arch x86_64` you are sending mixed
messages to your compiler. There are all manner of platform-specific problems
buried in there, that we don't need to worry about in the context of this
question.

_But let's just look at this line_ : `gfortran:f77:
build/src.macosx-10.7-intel-2.7/delaunay/sphere-f2pywrappers.f`

**what is numpy doing?!** I don't need any f2py features in this build! I
actually wrote a cython module _in order to avoid_ dealing with f2py's
insanity (I need to have 4 or 5 output variables, as well as neither-in-nor-
out arguments - neither of which is well supported in f2py.) I just want it to
compile `.c` -> `.o`, and `.f90` -> `.o` and link them. I could write this
compiler line myself if I knew how to include all the relevant headers.

Please tell me I don't need to write my own makefile for this... or that
there's a way to translate fortran to (output-compatible) C so I can just
avoid python ever seeing the .f90 extension (which fixes the whole problem.)
Note that [`f2c`](http://www.netlib.org/f2c/) is not suitable for this as it
only works on F77 and this is a more modern dialect (hence the `.f90` file
extension).

**UPDATE 2** The following bash script will happily compile and link the code
in place:

    
    
    PYTHON_H_LOCATION="/opt/local/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7/"
    
    cython sphere.pyx
    
    gcc -arch x86_64 -c sphere.c -I$PYTHON_H_LOCATION
    gfortran -arch x86_64 -c stripack.f90
    gfortran -arch x86_64 -bundle -undefined dynamic_lookup -L/opt/local/lib *.o -o sphere.so
    

Any advice on how to make this kind of hack compatible with a setup.py? I
don't anyone installing this module to have to go find `Python.h` manually...


Answer: **UPDATE:** I've created a project on github which wraps up this generating of
compile lines by hand. it's called [complicated_build](https://github.com/joe-
jordan/complicated_build).

**UPDATE 2:** in fact, "generating by hand" is a really bad idea as it's
platform specific -- the project now reads the values from the
`distutils.sysconfig` module, which is the settings used to compile python
(i.e. exactly what we want,) the only setting which is guessed is fortran
compiler and file extensions (which are user-configurable). I suspect it is
reimplementing a fair bit of distutils now!

* * *

The way to do this is to write your own compiler lines, and hack them into
your `setup.py`. I show an example below which works for my (very simple)
case, which has the following strucutre:

  * imports
  * `cythonize()` any `.pyx` files, so you only have fortran and C files.
  * define a `build()` function which compiles your code: 
    * maybe some easy-to-change constants, like compiler names and architecture
    * list up the fortran and C files
    * generate the shell commands that will build the modules
    * add the linker line
    * run the shell commands.
  * if the command was `install` and the target doesn't exist yet, build it.
  * run setup (which will build the pure python sections)
  * if the command was `build`, run the build now.

my implementation of this is shown below. It's designed for only one extension
module, and it recompiles all the files every time, so may require further
extension to be of more general use. Also note that I've hard coded various
unix `/`s, so if you're porting this to windows make sure you adapt or replace
with `os.path.sep`.

    
    
    from distutils.core import setup
    from distutils.sysconfig import get_python_inc
    from Cython.Build import cythonize
    import sys, os, shutil
    
    cythonize('delaunay/sphere.pyx')
    
    target = 'build/lib/delaunay/sphere.so'
    
    def build():
      fortran_compiler = 'gfortran'
      c_compiler = 'gcc'
      architecture = 'x86_64'
      python_h_location = get_python_inc()
      build_temp = 'build/custom_temp'
      global target
    
      try:
        shutil.rmtree(build_temp)
      except OSError:
        pass
    
      os.makedirs(build_temp) # if you get an error here, please ensure the build/ ...
      # folder is writable by this user.
    
      c_files = ['delaunay/sphere.c']
      fortran_files = ['delaunay/stripack.f90']
    
      c_compile_commands = []
    
      for cf in c_files:
        # use the path (sans /s), without the extension, as the object file name:
        components = os.path.split(cf)
        name = components[0].replace('/', '') + '.'.join(components[1].split('.')[:-1])
        c_compile_commands.append(
          c_compiler + ' -arch ' + architecture + ' -I' + python_h_location + ' -o ' +
          build_temp + '/' + name + '.o -c ' + cf
        )
    
      fortran_compile_commands = []
    
      for ff in fortran_files:
        # prefix with f in case of name collisions with c files:
        components = os.path.split(ff)
        name = components[0].replace('/', '') + 'f' + '.'.join(components[1].split('.')[:-1])
        fortran_compile_commands.append(
          fortran_compiler + ' -arch ' + architecture + ' -o ' + build_temp + 
          '/' + name + '.o -c ' + ff
        )
    
      commands = c_compile_commands + fortran_compile_commands + [
        fortran_compiler + ' -arch ' + architecture + 
        ' -bundle -undefined dynamic_lookup ' + build_temp + '/*.o -o ' + target
      ]
    
      for c in commands:
        os.system(c)
    
    
    if 'install' in sys.argv and not os.path.exists(target):
      try:
        os.makedirs('build/lib/delaunay')
      except OSError:
        # we don't care if the containing folder already exists.
        pass
      build()
    
    setup(
      name="delaunay",
      version="0.1",
      ...
      packages=["delaunay"]
    )
    
    if 'build' in sys.argv:
      build()
    

This could be wrapped up into a new `Extension` class I guess, with it's own
`build_ext` command - an exercise for the advanced student ;)

