Created By  dmarkovtsev Wednesday, April 11, 2018

When you do UI test automation and create a big enough test coverage the probability of having curtain portion of flaky tests increases. In many cases there is no reasonable way in terms of time and resources to mitigate all flakiness from the test coverage. In some cases even a flaky test may still be useful.

In this article we'll be collecting the best practices of dealing with flaky tests.

First we start from a set of methods you can apply to flaky tests without trying to fix them.

  1. Launch a test up to X times until first success.
  2. If X times from Y runs passed - consider the test passed.
  3. Quarantine (exclude from future runs) if probability of pass and failure is more than X percent after Y runs.
  4.  If status is the same as during previous run - ignore results.
  5.  Run and ignore status - gather statistics for future decision.
  6. Throw away. Test itself may be badly implemented.

There are different reasons for test flakiness and various techniques of fixing it. Below you can find links to some interesting articles about this topic.

 

Article Info
  • Last Updated: 4/11/2018
  • Article ID: KB336
  • Views: 299