Web applications are a perfect example of a rapidly moving target for UI test automation tools. Agile and DevOps drive small changes and frequent releases. This leads to certain challenges in QA field. How to create tests that survive after small UI changes? How to create tests that produce same results if ran 10 times in a row on an unmodified application? How to create tests that check exactly those features they were designed to check?
Answers to these questions are not entirely simple. Test automation is still an art.
In this blog we try to figure out what every test automation engineer can do to find the right answers. First I'll give a few examples that illustrate common challenges in Web UI test automation.
Assume that you have a page with
Log In button and the test relies on
xpath expression to find it. Imagine that a new element is inserted before this button. The xpath expression (selector) becomes invalid and the test breaks. Is it a good selector? I would say No. Can we do better? Sure, if the button has ID attribute specified we can use
With this selector no matter what is inserted/modified on the page - if
Log In button is there - the test will find it.
So, selectors matter.
Every Web application loads data from somewhere. It can be a database, third-party REST service or a banner network. And nobody knows how fast a web page is loaded and its elements are ready to use. So, an automated test must take into account various delays and be patient. Otherwise, the test may be useless as it will provide pass and fail results under the same circumstances.
Tests perform checks. Aristotle would call it
telos of tests. Let's compare two assertions:
Which one is better? Of course, the second assertion may lead to many confusions with the test. It will fail on a screen with a smaller resolution, it may produce a false positive result on a 5K Retina display.
The second assertion is weak. And the first one is strong.
A good test is a resilient test with strong assertions, it must meet the challenges presented above. Of course, there may be exceptions, but they are a minority.
What may help with building resilient tests for Web applications?
All modern browsers have private/incognito mode that makes you feel that you are running a web application for the first time. No cache, no cookies. It may help an automated test to enter the same water again and again and produce a steady result.
In Rapise - test automation tool that we develop - we have an XPATH generator integrated into Spy which automatically produces several xpath alternatives for a given DOM node. For example:
/html/body/form/div/div/div/p/input (//input) //input[@id="MainContent_LoginUser_LoginButton"] //input[@value='Log In']
By looking at xpath variants a test engineer can quickly choose the best one or produce a derivative.
It is nice to have an ability to specify a set of selectors that may be used to identify a particular element in the application. If some selectors are broken there is a chance that at least one is still valid.
Use wait statements preferably with a timeout to delay test execution util an element is available for interaction. Such waits can be explicit or implicit if, for example, a test engine tries to find an element on a page several times with a specific delay between lookup attempts.
Sometimes passing a click using DOM event does not work. In this case, it is useful to have a way to simulate a physical click.
Software developers may also help with building resilient tests. They can make life of QA teams much easier by
Building resilient tests is a key to successful UI test automation. If your tests are brittle better go with manual testing approach otherwise you'll spend too much time fixing those failing tests. Of course, right selection of an automation tool is also important. It should support you down the road of resiliency.