Greentea is the automated testing tool for Arm Mbed OS development. It's a test runner that automates the process of flashing development boards, starting tests and accumulating test results into test reports. You can use it for local development, as well as for automation in a continuous integration environment.
Greentea tests run on embedded devices, but Greentea also supports host tests. These are Python scripts that run on a computer and can communicate back to the embedded device. You can, for example, verify that a value wrote to the cloud when the device said it did.
This document will help you start using Greentea. Please see the htrun documentation, the tool Greentea uses to drive tests, for the technical details of the interactions between the platform and the host machine.
You can run tests throughout Mbed OS and for your project's code. They are located under a special directory called TESTS.
The fact that the code is located under this directory means that it is ignored when building applications and libraries. It is only used when building tests. This is important because all tests require a main() function, and building them with your application would cause multiple main() functions to be defined.
In this example, myproject is the project root, and all the source files under the test_case_1 directory are included in the test. The test build also includes any other source files from the OS, libraries and projects that apply to the target's configuration.
Note: You can name both the test group and the test case directory anything you like. However, you must name the TESTS directory TESTS for the tools to detect the test cases correctly.
Because test cases can exist throughout a project, the tools must find them in the project's file structure before building them.
Test discovery also obeys the same rules that are used when building your project. This means that tests that are placed under a directory with a prefix, such as TARGET_, TOOLCHAIN_ or FEATURE_, are only discovered, built and run if your current configuration matches this prefix.
This test case is only discovered if the target being tested supports the BLE feature. Otherwise, the test is ignored.
Generally, a test should not be placed under a TARGET_ or TOOLCHAIN_ directory because most tests should work for all target and toolchain configurations.
Tests can also be completely ignored by using the .mbedignore file described in the documentation.
A test case is named by its position in your project's file structure. For instance, in the above example, a test case with the path myproject/TESTS/test_group/test_case_1 would be named tests-test_group-test_case_1. The name is created by joining the directories that make up the path to the test case with a dash - character. This is a unique name to identify the test case. You will see this name throughout the build and testing process.
You can build tests through Arm Mbed CLI. For information on using Mbed CLI, please see the CLI documentation.
When you build tests for a target and a toolchain, the script first discovers the available tests and then builds them in parallel. You can also create a test specification file, which our testing tools can use to run automated hardware tests. For more information on the test specification file, please see the Greentea documentation.
The test.py script (not to be confused with tests.py), located under the tools directory, handles the process for building tests. It handles the discovery and building of all test cases for a target and toolchain.
Build the non-test code (all code not under a TESTS folder), but do not link it. The resulting object files are placed in the build directory.
Find all tests that match the given target and toolchain.
For each discovered test, build all of its source files and link it with the non-test code that was built in step 1.
If specified, create a test specification file and place it in the given directory for use by the testing tools. This is placed in the build directory by default when using Mbed CLI.
When building an Mbed application, the presence of an mbed_app.json file allows you to set or override different configuration settings from libraries and targets. However, because the tests share a common build, this can cause issues when tests have different configurations that affect the OS.
The build system looks for an mbed_app.json file in your shared project files (any directory not inside of a TESTS folder). If the system finds it, then this configuration file is used for both the non-test code and each test case inside your project's source tree. If there is more than one mbed_app.json file in the source tree, then the configuration system will error.
If you need to test with multiple configurations, then you can use the --app-config option. This overrides the search for an mbed_app.json file and uses the configuration file that you specify for the build.
You can write tests for your own project or add more tests to Mbed OS. You can write tests by using the Greentea client and the UNITY and utest frameworks, which are located in /features/frameworks.
By convention, all tests live in the TESTS/ directory. In the first-greentea-test folder, create a folder TESTS/test-group/simple-test/.
Tip: To see all tests, run mbed test --compile-list.
Change the test in a way that it fails (for example, expect 6 instead of 4), rerun the test and observe the difference.
The previous test was self-contained. Everything that ran only affected the microcontroller. However, typical test cases involve peripherals in the real world. This raises questions such as: Did my device actually get an internet connection, or did my device actually register with my cloud service? (We have a lot of these for Pelion Device Management.) To test these scenarios, you can use a host test that runs on your computer. After the device says it did something, you can verify that it happened and then pass or fail the test accordingly.
To interact with the host test from the device, you can use two functions: greentea_send_kv and greentea_parse_kv. The latter blocks until it gets a message back from the host.
This registers one function you can call from the device: init. The function checks whether the value was hello, and if so, returns world back to the device using the send_kv function.
You see the calls to and from the host through the greentea_send_kv and greentea_parse_kv functions. Note the GREENTEA_SETUP call. This specifies which host test to use, and the test is then automatically loaded when running (based on the Python name).
Debugging tests is a crucial part of the development and porting process. This section covers exporting the test and driving the test with the test tools while the target is attached to a debugger.
The easiest way to export a test is to copy the test's source code from its test directory to your project's root. This way, the tools treat it like a normal application.
You can find your exported project in the root project directory.
Assuming your test was exported correctly to your IDE, build the project and load it onto your target by using your debugger.
Bring the target out of reset and run the program. Your target waits for the test tools to send a synchronizing character string over the serial port. Do not run the mbed test commands because that will attempt to flash the device, which you've already done with your IDE.
Instead, you can use the underlying test tools to drive the test. htrun is the tool you need to use in this scenario. Installing the requirements for Mbed OS also installs htrun. You can also install htrun by running pip install mbed-host-tests.
From the output, take note of your target's serial port (in this case, it's COM270).
This detects your attached target and drives the test. If you need to rerun the test, reset the device with your debugger, run the program and run the same command.
For an explanation of the arguments used in this command, please run mbedhtrun --help.
This section highlights a few of the capabilities of the Greentea command-line interface. For a full list of the available options, please run mbed test --help.
After compilation, you can use the --run-list argument to list all tests that are ready to be ran.
The default action of Greentea using mbed test is to execute all tests found. You can also add -v to make the output more verbose.
When using the -n argument, you can use the * character as a wildcard. This command executes all tests that start with tests- and have -rtos- in them.
You can limit which boards Greentea uses for testing by using the --use-tids argument.
Where 02400203C3423E603EBEC3D8 and 024002031E031E6AE3FFE3D are the target IDs of platforms attached to your system.
You can view target IDs using mbed-ls, which is installed as part of Mbed CLI.
In this case, you won't test one target, the LPC1768.
Greentea supports a number of report formats.
This creates an interactive HTML page with test results and logs.
This creates an XML JUnit report, which you can use with popular Continuous Integration software, such as Jenkins.
This creates a general JSON report.
This creates a human-friendly text summary of the test run.
The Greentea test specification format decouples the tool from your build system. It provides important data, such as test names, paths to test binaries and the platform on which the binaries should run. This file is automatically generated when running tests through Mbed CLI, but you can also provide it yourself. This way you can control exactly which tests are run and through which compilers.
Greentea automatically looks for files called test_spec.json in your working directory. You can also use the --test-spec argument to direct Greentea to a specific test specification file.
When you use the -t / --target argument with the --test-spec argument, you can select which "build" to use. In the example below, you could provide the arguments --test-spec test_spec.json -t K64F-ARM to only run that build's tests.
Build K64F-ARM for NXP K64F platform compiled with ARMCC compiler.
Build K64F-GCC for NXP K64F platform compiled with GCC ARM compiler.
Place this file in your root folder, and name it test_spec.json.
There cannot be a main() function outside of a TESTS directory when building and running tests. This is because this function will be included in the nontest code build, as described in the building process section. When the test code is compiled and linked with the nontest code build, a linker error will occur, due to there being multiple main() functions defined. This is why you should either rename your main application file if you need to build and run tests, or use a different project. Note that this does not affect building projects or applications, only building and running tests.