Google Testing Blog: Flaky Tests at Google and How We Mitigate Them
Flaky Tests at Google and How We Mitigate Them
We have been dealing with this same phenomenon for years.Currently, we execute reliability runs of all of the CI tests (we try for hundreds of executions, but it depends on automation system load levels) per build to generate consistency rates. Using those numbers, we push product teams to move all tests that fall below a certain consistency level out of the CI tests. We keep them in the reliability suite for sake of coverage and issue discovery, but do not use them to gate submission into the main code branch.We likewise have difficulty accounting for the costs, but ballpark estimates show it is very expensive. I have done prior analysis to demonstrate that intermittent failures cause engineers to take longer to submit. Intermittent failures have a high duplicate bug rate, and ad hoc estimates from engineers is that we lose ~20 per duplicate bug for an engineer to determine there is duplication. The costs go way beyond all of that, though, particularly as process gates close down team productivity (failing CI tests lock a branch from changes until it is resolved), but also from legitimate bug escapes that were ignored because of the noise.It is my own opinion that even after tons of effort to reduce noise from tests, flaky tests are inevitable when the test condition reaches a certain complexity. There are more stable coding patterns (mostly in product, but also in test) which stabilize the test results, but they can only take you so far. Once you have moved the tests (e.g. convert end to end to unit tests, move pre-release tests to TIP methodologies) you still have a core set of test problems only discoverable in an integrated end to end system. And those tests will be flaky. If they are not flaky, they tend to never find bugs. This is not because the test is bad. It is because the conditions of the test, the thing that makes them flaky, are EXACTLY the same thing that caused the bug to be introduced in the first place. These bugs are scarier, riskier and harder to find. The secret, then, is to appropriately manage them. I prefer to rely more on repetition, statistics and runs that do not block the CI process. I prefer to data mine the test results and feed the work backlog.