Automated software testing is the ability to have a software tool or suite of software tools test your applications directly without human intervention. Generally test automation involves the testing tool send data to the application being tested and then compare the results with those that were expected when the test was created.
The average test plan for a commercial grade application will have between 2,000 and 10,000 test cases. Your test team of five must manually execute and document results for between 400 and 2,000 test cases. And the scheduled release date of your product is fast approaching. No worries; clone your team and work around the clock. Or perhaps there’s a better way.
As the graph above illustrates, there is an upfront cost to automated testing (as opposed to purely manual testing), but as the number of test cases and builds increases, the cost per test decreases.
This might be a good time to add automated testing to your test plan. The first step in this direction is realizing that no test plan can be executed completely with automated methods. The challenge is determining which components of the test plan belong in the manual test bucket and which components are better served in the automated bucket.
This is about setting realistic expectations; automation cannot do it all. You should not automate everything. Humans are smarter than machines (at least currently) and we can see patterns and detect failures intuitively in ways that computers are not able to.
Let’s begin by setting the expectations at a reasonable level. Let’s say we’ll automate 20% of test cases. Too small, management cries! Let’s address those concerns by describing what automating 20% means.
So we have decided to automate 20% of our test cases, great! There is only one problem - 20% means 20% of your test cases. How about the 20% of test cases used most often, that have the most impact on customer satisfaction, and that chew up around 70% of the test team’s time? The 20% of test cases that will reduce overall test time by the greatest factor, freeing the team for other tasks. That might be a good place to start.
These are the test cases that you dedicate many hours performing, every day, every release, every build. These are the test cases that you dread. It is like slamming your head into a brick wall – the outcome never seems to change. It is monotonous, it is boring, but, yes, it is very necessary. These test cases are critical because most of your clients use these paths to successfully complete tasks. Therefore, these are the tasks that pay the company and the test team to exist. These test cases are tedious but important.
Now that you have decided to add automated testing to your test plan, you need to consider which types of testing you will need:
The Unit testing part of a testing methodology is the testing of individual software modules or components that make up an application or system. These tests are usually written by the developers of the module and in a test-driven-development methodology (such as Agile, Scrum or XP) they are actually written before the module is created as part of the specification. Each module function is tested by a specific unit test fixture written in the same programming language as the module.
In an ideal world, the presentation layer would be very simple and with sufficient unit tests and other code-level tests (e.g. API testing if there are external application program interfaces (APIs)) you would have complete code coverage by just testing the business and data layers. Unfortunately, reality is never quite that simple and you often will need to test the Graphic User Interface (GUI) to cover all of the functionality and have complete test coverage. That is where GUI testing comes in.
Testing of the user interface (called a GUI when it’s graphics based vs. a simple text interface) is called GUI testing and allows you to test the functionality from a user’s perspective. Sometimes the internal functions of the system work correctly but the user interface doesn’t let a user perform the actions. Other types of testing would miss this failure, so GUI testing is good to have in addition to the other types.
API testing involves testing the application programming interfaces (APIs) directly and as part of integration testing to determine if they meet expectations for functionality, reliability, performance, and security. Since APIs lack a GUI, API testing is performed at the message layer. API testing is critical for automating testing because APIs now serve as the primary interface to application logic and because GUI tests are difficult to maintain with the short release cycles and frequent changes commonly used with Agile software development and DevOps.
When you release a new version of the system (e.g. changing some of the business components or internal data structures) you need to have a fast, easy to run set of API regression tests that verify that those internal changes did not break the API interfaces and therefore the client applications that rely on the APIs will continue to function as before.
There are several different types of performance testing in most testing methodologies, for example: performance testing is measuring how a system behaves under an increasing load (both numbers of users and data volumes), load testing is verifying that the system can operate at the required response times when subjected to its expected load, and stress testing is finding the failure point(s) in the system when the tested load exceeds that which it can support.