PROC. OF THE 6th EUR. CONF. ON PYTHON IN SCIENCE (EUROSCIPY 2013)

45

Performance of Python runtimes on a non-numeric
scientific code
Riccardo Murri∗†

arXiv:1404.6388v2 [cs.MS] 29 Apr 2014

F

Abstract—The Python library FatGHol [FatGHoL] used in [Murri2012] to reckon
the rational homology of the moduli space of Riemann surfaces is an example
of a non-numeric scientific code: most of the processing it does is generating
graphs (represented by complex Python objects) and computing their isomorphisms (a triple of Python lists; again a nested data structure). These operations
are repeated many times over: for example, the spaces M0,6 and M1,4 are
triangulated by 4’583’322 and 747’664 graphs, respectively.
This is an opportunity for every Python runtime to prove its strength in
optimization. The purpose of this experiment was to assess the maturity of
alternative Python runtimes, in terms of: compatibility with the language as
implemented in CPython 2.7, and performance speedup.
This paper compares the results and experiences from running FatGHol with
different Python runtimes: CPython 2.7.5, PyPy 2.1, Cython 0.19, Numba 0.11,
Nuitka 0.4.4 and Falcon.

Index Terms—python runtime, non-numeric, homology, fatgraphs

1

I NTRODUCTION

The moduli space Mg,n of smooth Riemann surfaces is a
topological space which has been subject of much research
both in algebraic geometry and in string theory. It is known
since the ’90s that this space has a triangulation indexed
by a special kind of graphs [Penner1988], [Kontsevich1992],
[ConantVogtmann2003], nicknamed "fat graphs".
Since graphs are combinatorial and discrete objects, a computational approach to the problem of computing topological
invariants of Mg,n is now feasible; algorithms to enumerate fatgraphs and compute their graph homology have been devised
in [Murri2012] and implemented in Python.
We propose an experiment whose purpose is to assess the
maturity of alternative Python runtimes, in terms of:
(a) compatibility with the language as implemented
in CPython 2.7, and
(b) performance speedup.
In particular, we were interested in the possible speedups
of a large non-numeric code.
* Corresponding author: riccardo.murri@gmail.com
† University of Zurich, Grid Computing Competence Center
c 2014 Riccardo Murri. This is an open-access article disCopyright ○
tributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in
any medium, provided the original author and source are credited.
http://creativecommons.org/licenses/by/3.0/

2

E XPERIMENT SETUP

The FatGHoL <http://fatghol.googlecode.com/> [FatGHoL]
program was used as a test code. FatGHoL computes
homology of the moduli spaces of Riemann surfaces
Mg,n via Penner-Kontsevich’s fatgraph simplicial complex
[Penner1988], [Kontsevich1992]. Homology is one of the most
important invariants of topological spaces. There are several
homology theories but they all share this computational procedure outline: given a vector space of (generalized) simplex
chains and a boundary operator, which is by definition a
linear operator D such that D2 = 0, the homology space is
by definition Ker D/ Im D. In graph homology, however, it
is the computation of these simplices and boundary that takes
up the largest fraction of compute time: the simplex chains
are defined as formal linear combinations of graphs, and the
boundary operator maps a graph into a linear combination
of graphs obtained by contracting its edges. Thus, explicit
construction of the simplices requires enumerating all distinct
isomorphism classes of fatgraphs, and then computing their
mutual relationships upon contraction of edges.
The FatGHoL program runs in three stages:
1. generate fatgraphs,
2. explicitly compute the boundary operator in matrix
form,
3. actually solve the homology linear system.
The last step has been disabled in the test code as it is
implemented in C++ for speed. What remains is 100% pure
Python code that runs on Python 2.6+ (but could run on 2.5
with minimal modifications).
FatGHoL involves a large number of graph isomorphism
computations: especially during fatgraph generation, each candidate fatgraph needs to be compared to all fatgraphs already
discovered, in order to avoid duplicates. In later stages, the
isomorphism computations are cached in memory, but in step
2 additional data is created for each graph, in order to pass
from fatgraphs to simplices.
It is worth noting that the FatGHoL code exercises many
of Python’s advanced data manipulation features, like list and
dictionary comprehensions, slicing, etc. but does not use any
kind of tight nested loops of the kind normally featured in
numeric codes.
Profiling data show more precisely how much work is done
at the Python level in the simplest case M0,4 . The following listing shows profiling data extracted from a complete

46

run of FatGHoL on CPython 2.7.5; 15787953 function calls
(15728052 primitive calls) were effected in 39.572 seconds;
the top 10 most called functions, ordered by call count are:

PROC. OF THE 6th EUR. CONF. ON PYTHON IN SCIENCE (EUROSCIPY 2013)

compiler. Given enough time, the JIT compiler gives
extremely good results, with speedups of 100% to 400%
relative to CPython in the M0,5 and M1,3 cases. In other
words, for the JIT approach to pay off, the code needs
ncalls
tottime filename:lineno(function)
2216088
2.175 rg.py:227(<genexpr>)
to perform many iterations of the same code path (this
966575
0.819 rg.py:143(is_loop)
is certainly the case for FatGHoL), because compiling
775362
0.839 cyclicseq.py:88(__getitem__)
a single function to native code takes a non-negligible
775362
0.634 rg.py:170(other_end)
722308
3.438 combinatorics.py:368(__init__)
amount of time. The break-even point for the FatGHoL
539039
1.689 cyclicseq.py:112(__getslice__)
code seems to be around 5 seconds of runtime: on M2,1 ,
506075/447917 0.745 cache.py:181(wrapper)
the CPU time taken by CPython and PyPy are almost
476134
1.122 combinatorics.py:441(rearranged)
385725
0.355 rg.py:137(__init__)
equal.
345740
0.849 rg.py:568(_first_unused_corner)
• Cython gives consistently about a 30% speedup on unmodified Python code. However, the "pure Python mode",
The FatGHoL code was run with seven different alternative
in which Cython takes variable typing hints embedded in
Python runtimes:
the code does not seem to give any advantage: results of
• CPython 2.7.5;
the two runs are not significantly different. This might
• Cython 0.19.1;
be related to a bug in the current version of Cython, see
• Cython 0.19.1 in "pure Python mode";
details in a later section.
• Falcon 0.05;
The
large memory consumption from PyPy and Nuitka
• Nuitka 0.4.4;
stands out in the memory data of figure 2. On the other
• PyPy 2.1;
hand, there is no significant increase in memory usage between
• Numba 0.10.0 and 0.11.0 with @autojit.
A detailed description of each of these is given in a later sec- CPython and Cython.
The large memory usage of PyPy can be explained by the
tion; Table 1 provides an overview of the installation and usage
fact
that the JIT infrastructure must keep in memory the profile
features of the different runtimes. The code used to install the
and
traces for all the code paths taken. In any long-running
software and run the experiments is available on GitHub at
program,
the memory should eventually reach a steady state
https://github.com/riccardomurri/python-runtimes-shootout.
and
not
increase
any further; it should be noted however,
Except for Cython in "pure Python mode" and Numba, all
that
in
these
benchmarks
the memory used by the PyPy JIT
runtimes run the unmodified Python code of FatGHoL. Cython
framework
dwarfs
the
memory
used by the program itself.
in "pure Python mode" requires the addition of decorators to
We
have
no
explanation
for
the
large memory consumption
the Python code that specify the types of function arguments
of
Nuitka.
and local variables to increase speedup of selected portions
of the code. Similarly, Numba uses the decorators @jit or
@autojit to mark functions that should be compiled to 4 R UNTIME SYSTEMS DETAILS
native code (the difference between the two decorators is 4.1 Cython 0.19.1 <http://cython.org/>
that that @autojit infers types at runtime, whereas @jit
Cython is a compiler for a superset of the Python language. It
requires the programmer to specify them1 ); we only used the
translates Python modules to a C or C++ source that is then
@autojit decorator to mark the same functions that were
compiled to a native code library that CPython can load and
marked as optimization candidates in the Cython experiment.
use. Cython optimizes best when users decorate the source
Each Python runtime was run on 4 test cases: computing the
code with hints at the types of variables and functions; it
homology of the M0,4 , M0,5 , M1,3 , and M2,1 moduli spaces. The
can also translate unmodified Python code, but then no type
test cases take from 0.20s to more than 2 minutes of runtime
inference is performed. Cython allows a variety of ways for
with CPython 2.7. Each test case was run 10 times and the best
giving these type hints; its so-called "pure Python" mode
time and lowest RAM occupation are reported in the summary
requires users to insert functions and variable decorators in
tables below.
the code: the Cython compiler can act on these directives, but
the CPython interpreter will instead load a cython module
3 R ESULTS
which turns them into no-ops.
Falcon and Numba could not run the code (see details in a
We tested Cython twice: on the unmodified Python sources,
later section) and thus do not appear in the report below.
and with hinting in the "pure Python" mode. The graphs show
For each runtime, the total used CPU time and memory were however very little difference between the two modes; this
measured: results and summary graphs are given in figures 1 could be a consequence of Cython defect ticket #477.
and 2. Detailed comparisons are given in the other figures.
Cython does its best when the source code is annotated
The CPU time data prompt a few observations:
with its extended keywords, which allow specifying the types
• PyPy gives the best results, provided the code runs
of variables (which allows optimizations, e.g., in loops), or
long enough to discount for the startup time of the JIT marking certain functions as C-only (which saves time when
dereferencing variables). This extended markup can be pro1. Note that in more recent versions of Numba, the two decorators have
vided either in the sources, or in additional .pxd files. We
been fused into one: @jit uses the supplied types, or infers them if none are
given.
have not done this exercise, however, as the amount of coding

PERFORMANCE OF PYTHON RUNTIMES ON A NON-NUMERIC SCIENTIFIC CODE

47

Runtime
Installed size (MB)

Cython 0.19.1
30 a

Falcon 0.05
14 a

Nuitka 0.4.4
25 a

Install script length
(SLOC)
Usage documentation

6

8

10

extensive

minimal

Porting/optimization
documentation

extensive

none

Porting/optimization
effort

none ("Pure Python"
mode) to very heavy
(.pxd hinting)

none: runs unmodified Python code

TABLE 1: Comparison of installation features of the Python runtimes.
b Does not need the CPython interpreter in addition, as all others do.

316s

5.12s
3.68s
3.68s
6.47s
5.25s

0.17s
0.14s
0.13s
0.15s
0.52s

0s

minimal, mostly examples

none

none

none: runs unmodified Python code

light
(w/
@autojit)
to
heavy (@jit with
types)

provides only a list
of compatibility issues; I could find
no list of Do-s and
Don’t-s for better
speed in PyPy
none: runs unmodified Python code

a

Plus 123MB for the CPython interpreter, which is anyway needed.

M0,4

M2,1

20.14s
14.54s
13.88s
22.25s
10.65s

10s
3s

short how-to to explain the different
compilation options
available
list of optimizations
that the runtime
does
(or
will)
support

19

M0,5

126.27s
93.25s
92.55s
135.48s
34.27s

31s

PyPy 2.1
162 b

CPU time of Python runtimes (synopsis)

PyPy 2.1
Nuitka 0.4.4
Cython 0.19.1
(w/ hints)
Cython 0.19.1
CPython 2.7.5

100s

Numba 0.11.0
97 a (+ 518MB of
LLVM 3.2)
24

M1,3

Fig. 1: Comparison of the total CPU time used by each runtime on the different test cases. The x-axis is sorted so that the runtimes for
CPython 2.7.5 are ascending. The y-axis shows values in seconds (smaller is better). Note that the y-axis is drawn on a logarithmic scale!

time required to properly mark all functions and variables is
quite substantial.
4.2

Falcon 0.05 <https://github.com/rjpower/falcon>

Falcon is a Python extension module that hacks into a CPython
interpreter and changes the execution loop, implementing
several optimizations (for instance, using a register-based VM
instead of a stack-based one) that the Falcon authors think
should be used in the upstream CPython interpreter too.
However, Falcon is still in early stages of development and
crashes on FatGHoL code with a segmentation fault.
4.3

Numba <http://numba.pydata.org/>

As its website states:
Numba is an optimizing compiler for Python; it
uses the LLVM compiler infrastructure to compile
Python syntax to machine code. It is NumPy-aware
and can speed up code using NumPy arrays. Other,
less well-typed code will be translated to Python CAPI calls effectively removing the "interpreter" but
not removing the dynamic indirection. Numba is also
not a Just-In-Time compiler.

Numba requires the code developer to use either the
@autojit (use run-time type info) or the @jit (explicitly
provide type information) decorators to mark those functions
that should be compiled. For our experiment, we used the
decorator @autojit on all functions that were decorated also
in the Cython test.
Versions 0.10.0 and 0.11.0 of Numba were tested; we could
not get either version to work.
Numba version 0.10.0 dies with an internal error ("TypeError: type_container() takes exactly 1 argument (3 given)",
reported as Issue #295 on Numba’s GitHub issue tracker), that
has been fixed in version 0.11.
However, Numba 0.11.0 with a "NotImplementedError:
Unable to cast from { i64, i8* }* to { i64, i8* }" message.
This has been reported as Issue #350 on the issue tracker and
is waiting for a fix.

4.4 Nuitka 0.4.4 <http://www.nuitka.net/>

Nuitka translates Python (2.6+) into a C++ program that then
uses libpython to execute in the same way as CPython
does, in a very compatible way. Although still in development,
Nuitka claims that it already:

48

PROC. OF THE 6th EUR. CONF. ON PYTHON IN SCIENCE (EUROSCIPY 2013)

Max used memory of Python runtimes (synopsis)

M2,1

261.16MiB
265.09MiB
264.67MiB
2445.69MiB
1632.53MiB

M0,4

191.89MiB
196.55MiB
196.67MiB
573.88MiB
709.55MiB

55.52MiB
61.22MiB
60.80MiB
229.02MiB
412.73MiB

39.03MiB
45.22MiB
44.83MiB
57.81MiB
217.75MiB

3162MiB
1000MiB
316MiB
100MiB
31MiB
10MiB
3MiB
0MiB

M0,5

M1,3

Fig. 2: Comparison of the total RAM used by each runtime on the different test cases. The x-axis is sorted so that the RAM usage for
CPython 2.7.5 are ascending. The y-axis shows values in MBs (smaller is better). Note that the y-axis is drawn on a logarithmic scale!

CPython 2.7.5
Cython 0.19.1
Cython 0.19.1
(w/ hints)
Nuitka 0.4.4
PyPy 2.1
0.0

CPU time of Python runtimes (M04)

Max used memory of Python runtimes (M04)

4th 0.17s
2nd 0.14s
1st 0.13s
3rd 0.15s
0.1

0.2

0.3

5th 0.52s

0.4

0.5

0.6

Fig. 3: Comparison of the total CPU time used by each runtime on
the M0,4 test case. The x-axis shows values in seconds.

CPython 2.7.5
Cython 0.19.1
Cython 0.19.1
(w/ hints)
Nuitka 0.4.4
PyPy 2.1
0

CPU time of Python runtimes (M05)

4th 20.14s

3rd 14.54s
2nd 13.88s
5th 22.25s
1st 10.65s

5

10

15

20

25

Fig. 4: Comparison of the total CPU time used by each runtime on
the M0,5 test case. The x-axis shows values in seconds.

CPython 2.7.5
Cython 0.19.1
Cython 0.19.1
(w/ hints)
Nuitka 0.4.4
PyPy 2.1
0

CPU time of Python runtimes (M13)

4th 126.27s

3rd 93.25s
2nd 92.55s
5th 135.48s
1st 34.27s

20

40

60

80

100 120 140 160

Fig. 5: Comparison of the total CPU time used by each runtime on
the M1,3 test case. The x-axis shows values in seconds.

CPython 2.7.5
Cython 0.19.1
Cython 0.19.1
(w/ hints)
Nuitka 0.4.4
PyPy 2.1
0

CPU time of Python runtimes (M21)
1st 3.68s
1st 3.68s
5th 6.47s
2

3

4

4th 5.25s
5

6

5th 217MiB

150

200

250

Fig. 7: Comparison of the total RAM usage by each runtime on the
M0,4 test case. The x-axis shows values in MBs.

CPython 2.7.5
Cython 0.19.1
Cython 0.19.1
(w/ hints)
Nuitka 0.4.4
PyPy 2.1
0

Max used memory of Python runtimes (M05)
1st 191MiB
2nd 196MiB
3rd 196MiB

4th 573MiB
5th 709MiB

100 200 300 400 500 600 700 800

Fig. 8: Comparison of the total RAM usage by each runtime on the
M0,5 test case. The x-axis shows values in MBs.

Max used memory of Python runtimes (M13)

CPython 2.7.5 1st 261MiB
Cython 0.19.1 3rd 265MiB
Cython 0.19.1 2nd 264MiB
(w/ hints)
Nuitka 0.4.4
PyPy 2.1
0
500

5th 2445MiB
4th 1632MiB

1000

1500

2000

2500

3000

Fig. 9: Comparison of the total RAM usage by each runtime on the
M1,3 test case. The x-axis shows values in MBs.

Max used memory of Python runtimes (M21)

3rd 5.12s

1

CPython 2.7.5 1st 39MiB
Cython 0.19.1
3rd 45MiB
Cython 0.19.1 2nd 44MiB
(w/ hints)
Nuitka 0.4.4
4th 57MiB
PyPy 2.1
0
50
100

7

Fig. 6: Comparison of the total CPU time used by each runtime on
the M2,1 test case. The x-axis shows values in seconds.

CPython 2.7.5 1st 55MiB
Cython 0.19.1 3rd 61MiB
Cython 0.19.1 2nd 60MiB
(w/ hints)
Nuitka 0.4.4
PyPy 2.1
0
100

4th 229MiB
200

300

5th 412MiB
400

Fig. 10: Comparison of the total RAM usage by each runtime on the
M2,1 test case. The x-axis shows values in MBs.

PERFORMANCE OF PYTHON RUNTIMES ON A NON-NUMERIC SCIENTIFIC CODE

create[s] the most efficient native code from this.
This means to be fast with the basic Python object
handling.
Results of this experiment do not seem to corroborate this
claim.
4.5

PyPy 2.1 <http://pypy.org/>

PyPy is a Python language interpreter with a Just-In-Time
compiler (and many other features!). It can thus translate
repetitive Python code into native code on the fly. PyPy must
first be bootstrapped by compiling itself, which takes a lot of
time and RAM, but then it is a drop-in replacement for the
python command and just works.

5

ACKNOWLEDGEMENTS

The author acknowledges support of the Informatik Dienste
of the University of Zurich, particularly for the usage of
the new SGI UV 2000 machine for running the tests. I
would also like to thank Kay Hayen, Marc Florisson, Russel
Power and Alex Rubynstein for their readiness to discuss
and fix the bugs I reported on Nuitka, Numba, and Falcon.
Finally, I would like to express my gratitude to all those
who made remarks and inquiries at the EuroSciPy poster
session, and particularly Ronan Lamy and Denis Engemann
for their insightful comments. Finally, I would like to thank
Mike Mueller and Pierre de Buyl for reviewing the initial draft
paper and making many useful suggestions for improving it.

R EFERENCES
[Murri2012]

R. Murri. Fatgraph Algorithms and the Homology of the Kontsevich Complex, arXiv preprint
arXiv:1202.1820, February 2012.
[FatGHoL]
R. Murri. The FatGHoL software website, http://
fatghol.googlecode.com/
[Penner1988]
R. C. Penner. Perturbative series and the moduli
space of Riemann surfaces, J. Differential Geom,
1988.
[Kontsevich1992]
M. Kontsevich. Formal (non)-commutative symplectic geometry, The Gelfand Mathematical Seminars,
1990–1992.
[ConantVogtmann2003] J. Conant, K. Vogtmann. On a theorem of Kontsevich, Algebr. Geom. Topol., 2003.

49

50

PROC. OF THE 6th EUR. CONF. ON PYTHON IN SCIENCE (EUROSCIPY 2013)

