How to do continuous integration with gitlab when developing blender addons?
Introduction

Continuous integration requires a few things, namely tests, a CI server and tooling to run the test suite and read results automatically. Let's take them one by one.

Testing

There are various ways to write unit tests in python, but I decided to go with the classic unittest module because it is included with blender's python.

While I started off with tests in a different repository, I decided to move them alongside the addon in order to eliminate the need of keeping tests and addon in sync.

Let's say our addon is named toaster. I would advise naming the tests folder toaster_tests. You'll see why.

The test suite

The test suite is written in classic unittest manner.

class ToasterMeshTests(unittest.TestCase):
    def setUp(self):
        if bpy.data.objects.get('Cube') is not None:
            bpy.data.objects.remove(bpy.data.objects['Cube'], do_unlink=True)

    def tearDown(self):
        pass

    def test_addon_enabled(self):
        self.assertIsNotNone(toaster.bl_info)


Some caveats:


You need a way to override context because the tests are not always run in the same context as the actual operators. I have found this to mostly work but not always.. Some times you will have to rework your addon in a way that it is less context-sensitive, for example using bmesh instead of operators to manipulate meshes. You will also need to provide a path for context passing all the way down to the last operator call (which is a good idea anyway as you might find that things break due to context changes when you move your UI around). This context override function seems to work most of the time for me (unlike the one in the API reference)

def context_override(self):
    override = bpy.context.copy()
    for window in bpy.context.window_manager.windows:
        screen = window.screen
        for area in screen.areas:
            if area.type == 'VIEW_3D':
                override['window'] = window
                override['screen'] = screen
                override['area'] = area
                override['region'] = [region for region in area.regions if region.type == 'WINDOW'][0]
                override['selected_objects'] = [o for o in bpy.context.scene.objects if o.select]
                override['active_object'] = bpy.context.scene.objects.active
                return override

I found a weird bug when reloading the startup file (bpy.ops.wm.read_homefile()) in setUp so for now you need to manually start the suite in an empty file, and clean up after each test.


Graphical version

If your addon is graphical it might make sense to run some of the tests manually and eyeball the result instead of writing complex geometry-understanding scripts. For this reason, I made the tests folder an addon by itself, with a panel and an operator that calls each test separately. This is why I recommend naming the folder toaster-tests, in order to avoid name conflicts in the blender addons folder.

This is the operator

class TestsOperator(bpy.types.Operator):
    bl_idname = "toaster_tests.run"
    bl_label = "Toaster Tests"
    bl_options = {'REGISTER', 'UNDO'}
    testcase = bpy.props.StringProperty()

    def execute(self, context):
        if self.testcase == "all":
            suite = unittest.defaultTestLoader.loadTestsFromTestCase(ToasterMeshTests)
        else:
            suite = unittest.TestSuite()
            suite.addTest(ToasterMeshTests(self.testcase))
        unittest.TextTestRunner().run(suite)
        return {'FINISHED'}


And the UI panel

#A new panel in the tools section of the 3D view
class View3DPanel():
    bl_space_type = 'VIEW_3D'
    bl_region_type = 'TOOLS'


class ToasterTestsPanel(View3DPanel, Panel):
    bl_label = "Toaster Tests"
    bl_context = "objectmode"
    bl_category = "Toaster Tests"

    def draw(self, context):
        layout = self.layout
        col = layout.column(align=True)
        col.operator("toaster_tests.run").testcase = "all"
        col = layout.column(align=True)
        col.operator("toaster_tests.run", text="Test 1").testcase = "test_add_column"
        col.operator("toaster_tests.run", text="Test 2").testcase = "test_add_beam"




Running the test from the console

To enable continuous integration you need to be able to run the tests from the terminal. While this is trivial, it is not as trivial to make sure that the CI runner understands if the tests failed or passed. Blender has the --python-exit-code parameter but this returns non-zero only if an exception is thrown in the supplied python file, immediately before exit.

Thus we need to throw an exception if the tests fail, catch it and immediately exit. This is what this script does.

import unittest
from toaster_tests.suite_1 import ToasterMeshTests
import sys

def run():
    suite = unittest.defaultTestLoader.loadTestsFromTestCase(ToasterMeshTests)
    success = unittest.TextTestRunner().run(suite).wasSuccessful()
    if not success:
        raise Exception('Tests Failed')

try:
    run()
    sys.exit(0)
except Exception:
    sys.exit(1)


You can try to run it like this

blender --addons toaster,toaster_tests --python toaster_tests/run.py --python-exit-code 1


You will see blender momentarily launch and output like this:

Read new prefs: /path/to/userpref.blend
Toaster Loaded
Reloaded modules
...
----------------------------------------------------------------------
Ran 3 tests in 0.003s

OK


or if the tests failed

Read new prefs: /path/to/userpref.blend
Toaster Loaded
Reloaded modules

.F.
======================================================================
FAIL: test_add_column (toaster_tests.suite_1.ToasterMeshTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/qwazix/.config/blender/2.78/scripts/addons/toaster_tests/suite_1.py", line 45, in test_add_column
    self.assertEquals(bpy.data.objects['Cube'].location, Vector((0,0,-8)), "Column not at the correct position")
AssertionError: Vector((0.0, 0.0, 0.0)) != Vector((0.0, 0.0, -8.0)) : Column not at the correct position

----------------------------------------------------------------------
Ran 3 tests in 0.002s

FAILED (failures=1)


CI!

In the first reference link there's a nice writeup on how to make the CI server (in that case Travis) to download blender in the new VM the CI engine uses. While this allows you to use the shared runners in the case of GitLab (I assume something similar happens with Travis) however I find it wasteful and unnecessarily heavy on blender foundation's servers to download blender each time I push to my server, so I decided to spin up a virtual machine on my local computer.

So go install yourself a distro with your favorite blender version on a virtual machine and then install gitlab-ci-multi-runner. 


  If you are wondering why a VM, it's because you might have done all sorts of weird things on your local blender installation (enabled multiple addons, installed python modules and other things) that a user might not have and thus the fact that the tests pass on your machine doesn't mean it will work on a clean environment.


Register your runner with gitlab 

gitlab-runner register


It will ask you to provide a token from your gitlab repository and give a name to the runner. For the first time it might be useful to do it as user rather than as root, in order to run it manually and be able to observe what is going on. Afterwards, registering as root allows for unattended tests by just having the VM booted up, though you might run into issues with the X11 display. Choose a name and labels as you like and set executor as shell.

Now run it (except if you set up as root, where it automatically runs in the background)

gitlab-runner run


CI configuration

To trigger a CI build in GitLab you need to add a .gitlab-ci.yml in the root of your repository. The following file works for me

job1:

  script:
    - mkdir -p ~/.config/blender/2.78/scripts/addons/
    - rm -f ~/.config/blender/2.78/scripts/addons/toaster
    - rm -f ~/.config/blender/2.78/scripts/addons/toaster_tests
    - ln -s `pwd`/toaster/ ~/.config/blender/2.78/scripts/addons
    - ln -s `pwd`/toaster_tests/ ~/.config/blender/2.78/scripts/addons
    - blender --addons toaster,toaster_tests --python toaster_tests/run.py --python-exit-code 1 


If you choose to run it as root you will need to append the following switches to the last line, as the runner runs under the gitlab-runner user which does not have a DISPLAY, nor access to alsa.

-b -noaudio



  Note that I suspect that running with -b might incur additional context issues with blender, since it's possible that context override won't work (no windows to loop through in context_override()). This theory is not tested.


Passed!

Now if all went well, your runner should show 

Checking for builds... received                     build=6237996 repo_url=https://gitlab.com/d-e/blender-addon-boilerplate.git runner=78a379b6


Build succeeded

and the build page in gitlab should show the output from blender. After a while a nice green reassuring



badge should be besides your last commit.

Wrapped and ready to go

I've updated my boilerplate blender addon (now named Toaster) with CI configuration. There are two branches, one that launches the full blender gui in the runner and the nogui-testing branch which runs blender in the background. Go get it.

References


Automatic Blender Addon Testing with TravisCI
Terminate Blender with Exit Code “1” when run from command line
Blender Manual &ndash; Command Line Arguments
Gitlab runner documentation