Upgrading to Python 3 with Zero Downtime

by Adam Johnson

We recently upgraded our 160,000 lines of backend Python code from Python 2 to Python 3. We did with zero downtime and no major errors! Here's how we did it, hopefully it will help anyone else still stuck on Python 2!

This was basically a three step process:

Make the code compatible with both Python 2 and 3 Deploy Clean up the Python 2 compatibility code, and use some Python 3 idioms

That makes it sound simple, but each step was a little more involved… I'll expand on each in sections below.

Make the code compatible with both Python 2 and 3

Since we operate on a continuous delivery schedule, we couldn't just stop everything, convert the code base over to Python 3, deploy, and continue. Rather, we made the code compatible with both Python versions whilst new features were being written, allowing us to roll over to Python 3 when ready, and even roll back to Python 2 when needed, without rolling back features.

Making our dependencies compatible

The first step to make our code base compatible was to ensure that all our dependencies were! We have nearly 200 of them at this point, so auditing them all took some time. Luckily there's the useful caniusepython3 tool which checks for the appropriate PyPI classifiers on your installed packages. We found some cases where packages were just missing the correct classifiers, or where the Python 3 compatibility had been recently released recently and we needed to upgrade, but there were about twenty packages were just not compatible and needed fixing.

We got to work making pull requests. Unfortunately in a few cases the original libraries were totally abandoned; we found for them that we were only using a fraction of the library so porting it all ourselves didn't look attractive, and thus we made our own to fill in the gaps (e.g. treepoem and poor-smime-sign ).

The other thing to look out for is where forks already exist for Python 3, but the original package is abandoned and hasn't been updated to link to the fork. The main one for us was the abandoned python-openid which has been forked to python3-openid - it was only by an almost accidental web search that we found out about this! Because of this, we ended up with two requirements files, one for each python version, and a base file that they both included with the line -r requirements-base.txt .

After all the dependencies were confirmed compatible, we started adding backports packages on Python 2 to use some Python 3 features and compatibility done early. For example, backports.csv is very useful for getting the Python 3 Unicode-friendliness of the csv module on Python 2 - we'd actually already written a wrapper around the Python 2 csv and could delete that just by using this library.

Making our code compatible

Once all that was done, we had to make sure our own code was Python 3 ready. For a long time we'd been adding all the __future__ headers possible on Python 2 ( absolute_import, division, print_function, unicode_literals ) as they all helped with preventing bugs. We do this with excellent isort plus this config in our setup.cfg :

[isort] add_imports = from __future__ import absolute_import from __future__ import division from __future__ import print_function from __future__ import unicode_literals

We had also been forcing all files to be UTF-8 for a long time, using the encoding header # -*- encoding:utf-8 -*- , enforced by flake8-coding with this config in setup.cfg :

[flake8] ; flake8-coding accept-encodings = utf-8

To get to fully cross-compatible code, we added another linter to our toolchain: python-modernize . This uses the power of 2to3 , but rather than just convert once, it moves your code to be compatible with both Pythons, via the amazing six .

We activated modernize one rule at a time, every few days, starting with the easier ones and working our way up. This allowed everyone to be synchronized on the compatible style by learning a few rules at a time, e.g. "use six.text_type instead of unicode ", rather than getting them all at once.

Another thing that we found important was ensuring that the six.moves aliases and the backport packages we added were used. To help with this I added a new option to my flake8 plugin flake8-tidy-imports and then added some rules on the YPlan code base, which were configured in our setup.cfg like so:

[flake8] ; flake8-tidy-imports banned