Using Mozmill for testing Firefox addon
Question: I have a Mozilla Firefox addon and wanna test it. I've found Mozmill and wrote
the small Python script, which is just taking each Firefox version and run the
command like this:

    
    
    mozmill --binary=C:\browsers\firefox\38.0.1\firefox.exe 
    --addon=C:\my_ext\ext 
    --test=C:\my_ext\tests\mozmill\
    

Here is the script `unit_test_runner_public.py`:

    
    
    #!/usr/bin/env python
    # -*- coding: utf-8 -*-
    
    from subprocess import check_output
    import time
    
    firefox_binary_path = 'C:\\browsers\\firefox\\'
    ff_addon_location = 'C:\\my_ext\\ext'
    tests_folder = 'C:\\my_ext\\tests\\mozmill\\'
    ff_versions = ['38.0.1', '37.0.2', '36.0.4', '35.0.1', '34.0.5', '33.1.1',
                   '32.0.3', '31.0', '30.0', '29.0.1', '28.0', '27.0.1', '26.0',
                   '25.0.1', '24.0', '23.0.1', '22.0', '21.0', '20.0.1', '19.0.2',
                   '18.0.2', '17.0.1', '16.0.2', '15.0.1', '14.0.1', '13.0.1']
    
    
    def build_ff_path(ff_version):
        return "%s%s%s" % (firefox_binary_path, ff_version, "\\firefox.exe")
    
    
    for item in ff_versions:
        print "##### Started unit tests for Mozilla Firefox %s #####" % item
        current_run = "mozmill --binary=%s --addon=%s --test=%s" % \
            (build_ff_path(item), ff_addon_location, tests_folder)
        test_run_result = check_output(current_run, shell=True)
        print "##### Finished unit tests for Mozilla Firefox %s #####" % item
    
        time.sleep(10)
    

So Mozmill is starting the browser, running the tests and then closing the
browser and doing it for each Firefox version starting from 38.0.1 to 13.0.1

The problem is, that almost each time it hangs on some random Firefox version.
So it opens the browser instance, run the tests, but then it's not closing the
browser and the Firefox window hangs for some time and then I see such
exception in the terminal:

    
    
    ##### Finished unit tests for Mozilla Firefox 16.0.2 #####
    ##### Started unit tests for Mozilla Firefox 15.0.1 #####
    mozversion INFO | application_buildid: 20120905151427
    mozversion INFO | application_changeset: 0b774a1067fe
    mozversion INFO | application_display_name: Firefox
    mozversion INFO | application_id: {ec8030f7-c20a-464f-9b0e-13a3a9e97384}
    mozversion INFO | application_name: Firefox
    mozversion INFO | application_repository: http://hg.mozilla.org/releases/mozilla
    -release
    mozversion INFO | application_vendor: Mozilla
    mozversion INFO | application_version: 15.0.1
    mozversion INFO | platform_buildid: 20120905151427
    mozversion INFO | platform_changeset: 0b774a1067fe
    mozversion INFO | platform_repository: http://hg.mozilla.org/releases/mozilla-re
    lease
    mozversion INFO | platform_version: 15.0.1
    Traceback (most recent call last):
      File "C:\Python27\lib\site-packages\mozmill\__init__.py", line 878, in run
        mozmill.run(tests, self.options.restart)
      File "C:\Python27\lib\site-packages\mozmill\__init__.py", line 473, in run
        self.stop_runner()
      File "C:\Python27\lib\site-packages\mozmill\__init__.py", line 595, in stop_ru
    nner
        raise Exception('client process shutdown unsuccessful')
    Exception: client process shutdown unsuccessful
    Traceback (most recent call last):
      File "unit_test_runner_public.py", line 24, in <module>
        test_run_result = check_output(current_run, shell=True)
      File "C:\Python27\lib\subprocess.py", line 573, in check_output
        raise CalledProcessError(retcode, cmd, output=output)
    subprocess.CalledProcessError: Command 'mozmill --binary=C:\browsers\firefox\15.
    0.1\firefox.exe --addon=C:\my_ext\ext --test=C:\my_ext\tests\mozmill\' returned 
    non-zero exit status 1
    
    C:\my_ext>
    

And every time it happens with the random version of Firefox, so there is no
pattern like issue with some specific Firefox version.

System details are:

  * OS: Microsoft Windows 7 Enterprise SP1 x86 
  * Python: 2.7.9 
  * Mozmill: 2.0.10

And the output of pip list:

    
    
    blessings (1.6)
    jsbridge (3.0.3)
    ManifestDestiny (0.5.7)
    manifestparser (1.1)
    mozcrash (0.14)
    mozdevice (0.45)
    mozfile (1.1)
    mozinfo (0.7)
    mozlog (2.11)
    mozmill (2.0.10)
    moznetwork (0.24)
    mozprocess (0.22)
    mozprofile (0.23)
    mozrunner (5.35)
    mozversion (1.0)
    pip (1.5.6)
    setuptools (7.0)
    

Does anyone had experience with such issues?


Answer: So, after posting a question on [Mozmill Developers Google
Groups](https://groups.google.com/forum/#!topic/mozmill-dev/6BdIJtoWVqw) \-
I've got an answer from [Henrik Skupin](https://github.com/whimboo), who is
responsible for Mozmill.

In short, they've also came across this issue, but Mozmill will be
discontinued soon - and it's better start using the new framework [firefox-ui-
tests](https://github.com/mozilla/firefox-ui-tests). Unfortunately it doesn't
have yet possibility to specify `--addon` like it was with Mozmill, but
[bug](https://bugzilla.mozilla.org/show_bug.cgi?id=1168803) (feature request)
;) is already created - so most likely this functionality will be added in
future releases. Meanwhile, if you need to test Firefox addon, you can specify
`--profile` to already prebuilt Firefox profile with preinstalled addon in it.

