Showing test count in buildbot
Question: I am not particularly happy about the [stats that Buildbot
provides](http://buildbot.scons.org/builders/centos7-python-2.7/builds/5). I
understand that it is for building and not testing - that's why it has a
concept of Steps, but no concept of Test. Still there are many cases when you
need test statistics from build results. For example when comparing skipped
and failed tests on different platforms to estimate the impact of a change.

So, what is needed to make Buildbot display test count in results?  
What is the most simple way, so that a person who don't know anything about
Buildbot can do this in 15 minutes?


Answer: Depending how you want to process the test results and how the test results
are presented, Buildbot does provide a `Test` step,
`buildbot.steps.shell.Test`

An example of how I use it for my build environment:

    
    
    from buildbot.steps import shell
    
    class CustomStepResult(shell.Test):
        description = 'Analyzing results'
        descriptionDone = 'Results analyzed'
    
        def __init__(self, log_file = None, *args, **kwargs):
            self._log_file = log_file
            shell.Test.__init__(self, *args, **kwargs)
            self.addFactoryArguments(log_file = log_file)
    
        def start(self):
            if not os.path.exists(self._log_file):
                self.finished(results.FAILURE)
                self.step_status.setText('TestResult XML file not found !')
            else:
                import xml.etree.ElementTree as etree
                tree = etree.parse(self._log_file)
                root = tree.getroot()
    
                passing = len(root.findall('./testsuite/testcase/success'))
                skipped = len(root.findall('./testsuite/testcase/skip'))
                fails = len(root.findall('./testsuite/error')) + len(root.findall('./testsuite/testcase/error')) + len(root.findall('./testsuite/testcase/failure'))
    
                self.setTestResults(total = fails+passing+skipped, failed = fails, passed = passing)
                ## the final status for WARNINGS is green but the step itself will be orange
                self.finished(results.SUCCESS if fails == 0 else results.WARNINGS)
                self.step_status.setText(self.describe(True))
    

And in the configuration factory I create a step as below:

    
    
    factory.addStep(CustomStepResult(log_file = log_file))
    

Basically I override the default `Test` shell step and pass a custom XML file
which contains my test results. I then look for the `pass/fail/skip` result
nodes and accordingly display the results in the waterfall.

![Sample Test step with the statistics](http://i.stack.imgur.com/5fg3J.jpg)

