How to test provisioning and configuration in Ansible setup?
Some options out there..

Testing tools: Sorted by github stars


Serverspec - Ruby, most popular tool out there, built on ruby's rspec
Goss - YAML, simple, &lt;10MB self-contained binary, extremely fast, can generate tests from system state
Inspec - Ruby, think of it as an improved serverspec, almost same syntax, made by the chef guys. Built to be easier to extend than serverspec
Testinfra - Python, has the cool feature of being able to use Ansible's inventory/vars


Major differences between them:

Ultimately, I would suggest spending a day experimenting with all of them to get a feel for them before deciding for yourself.


With the exception of Goss, all the frameworks can run against a remote machine (ex. over ssh). Goss only runs locally or in docker w/ dgoss.
All frameworks can be run locally on a server, but require Python or Ruby to be installed or embedded. Inspec provides a self-contained &lt;150MB bundle with an embedded version of ruby. Goss is a single &lt;10MB binary with no external dependencies.
Goss has built in support for nagios/sensu output, this allows for easier integration with monitoring tools.
Goss tests tend to be simpler, but less flexible since it's based on YAML. Other frameworks allow you to leverage the full power of the underlying language Python/Ruby to write tests or extend the tool's functionality. (simplicity vs flexibility)
Goss allows you to generate tests from current system state
Testinfra to my knowledge is the only one that has built-in support for ansible inventory and variables
Inspec is backed by Chef


Continuous/divergence testing:


Chef Compliance - works with inspec to continuously test your servers, paid product
Goss - Can be easily hooked into Nagios or Sensu. Also, supports exposing server tests as an http endpoint.


Testing harnesses for development:


kitchen - Testing harness tool, launches instance, runs config management code, runs test suite. Made by the chef guys
Molecule - Similar to test kitchen, but written specifically for ansible


Full Disclosure: I'm the author of goss

UPDATE: InSpec 4.x or above uses a mixed commercial / open source license - see comments.