Apply resilience-testing in a company (to promote DevOps)? If true, why, how and what effect does it have on the Engineers? If false, why not? Back in the day this Q&amp;A was created.

ChaosMonkey is a tool that is able to stop random services on a platform in a certain time period. The goal of this is to get a stable platform that is fault tolerant. If this happens during the day when most of the Engineers are working, they get proactive and the infrastructure will be hardened.

At the moment, more and more services are created in AWS, GCP and Azure. The services on these platforms are getting more and more critical. Now it is time to onboard more and more engineers. Especially most of the developers are still silo minded, i.e. some of them only want to develop, but this is shifting drop by drop.

According to Wikipedia, ChaosMonkey was invented at Netflix and additional resilience tools were created as well:

| Purpose                                        | Tool              |
|:----------------------------------------------:|:-----------------:|
| Stops random services in production randomly   | Chaos Monkey      |
| Drop an availability zone, e.g. in AWS         | Chaos Gorilla     |
| Communication delays                           | Latency Monkey    |
| Fix unhealthy services                         | Doctor Monkey     |
| Dispose unused resources                       | Janitor Monkey    |
| Detect nonconforming instance and notify owner | Conformity Monkey |
| Disable instances that are vulnerable          | Security Monkey   |
| Detect problems between 10-18                  | 10-18 Monkey      |


Based on this table one could also achieve also the same purpose by start to write some shell scripts that stop random services for example.

Based on the information that was read, the definition of resilience-testing is basically testing if an app and underlying infrastructure is error prone in production.

This is also confirmed by this blog post that refers to how resilience testing is done at AWS, but also describes IBMs view on this type of testing. According to IBM there are two parts that have to be taken into account, i.e. problem impact and service level. In an ideal world the customer will not notice any downtime at all, but if a machine dies (minutes) or a complete data center (hours) then the customer could experience downtime. The most important thing to be transparently as possible to the customer and to minimize impact.

Currently, resilience-testing is not used in the company, but the more is read about it, it becomes interesting. Especially as it is also aligned with DevOps and the company tries to make Engineers DevOps minded. It looks like that resilience-testing promotes a DevOps mindset.

Resilience testing is DevOps related

Some people could argue that testing is not part of DevOps, but according to Wikipedia, tools like ChaosMonkey are aligned with the DevOps toolchain.
I think that chaos testing will promote DevOps unless people have been informed in advance and proper alert management has been enabled. If people do not get triggered then this will not work.