In software testing, test automation (which includes automated test execution) is one or more of the following tasks: Using purpose built software tools to control and set up test preconditions Executing tests Comparing actual outcomes to predicted outcomes. — ISTQB1
Test automation helps running many test cases consistently and repeatedly on different versions of the SUT and/or environments. It is not only a mechanism for running a test suite without human interaction, it also involves a process of designing the testware, including:
- Test cases
- Test environments
- Test data
Testware is needed for the testing activities which include:
- Implementing automated test cases
- Monitoring and controlling the execution of automated tests
- Interpreting, reporting and logging the automated test results
Test automation can interact in different ways with a SUT:
- Testing through the public interfaces to classes, modules or libraries of the SUT (API testing)
- Testing through the user interface of the SUT (e.g., GUI testing or CLI testing)
- Testing through a service or protocol
Objectives of test automation include:
- Improving test efficiency
- Providing wider function coverage
- Reducing the total test cost
- Performing tests that manual testers cannot
- Shortening the test execution period
- Increasing the test frequency/reducing the time required for test cycles
Advantages of test automation include:
- More tests can be run per build
- The possibility to create tests that cannot be done manually (real-time, remote, parallel tests)
- Tests can be more complex
- Tests run faster
- Tests are less subject to operator error
- More effective and efficient use of testing resources
- Quicker feedback regarding software quality
- Improved system reliability (e.g., repeatability, consistency)
- Improved consistency of tests
Disadvantages of test automation include:
- Additional costs are involved
- Initial investment to setup TAS
- Requires additional technologies
- Team needs to have development and automation skills
- On-going TAS maintenance requirement
- Can distract from testing objectives, e.g., focusing on automating tests cases at the expense of executing tests
- Tests can become more complex
- Additional errors may be introduced by automation
Limitations of test automation include:
- Not all manual tests can be automated
- The automation can only check machine-interpretable results
- The automation can only check actual results that can be verified by an automated test oracle
- Not a replacement for exploratory testing
Success Factors in Test Automation
The following success factors apply to test automation projects that are in operation and therefore the focus is on influences that impact on the long term success of the project. Factors influencing the success of test automation projects at the pilot stage are not considered here.
Test Automation Architecture (TAA)
The Test Automation Architecture (TAA) is very closely aligned with the architecture of a software product. It should be clear which functional and non-functional requirements the architecture is to support. Typically this will be the most important requirements. Often TAA is designed for maintainability, performance and learnability. It is helpful to involve software engineers who understand the architecture of the SUT.
The SUT needs to be designed for testability that supports automated testing. In the case of GUI testing, this could mean that the SUT should decouple as much as possible the GUI interaction and data from the appearance of the graphical interface. In the case of API testing, this could mean that more classes, modules or the command-line interface need to be exposed as public so that they can be tested. The testable parts of the SUT should be targeted first. Generally, a key factor in the success of test automation lies in the ease of implementing automated test scripts. With this goal in mind, and also to provide a successful proof of concept, the Test Automation Engineer (TAE) needs to identify modules or components of the SUT that are easily tested with automation and start from there.
Test Automation Strategy
A practical and consistent test automation strategy that addresses maintainability and consistency of the SUT. It may not be possible to apply the test automation strategy in the same way to both old and new parts of the SUT. When creating the automation strategy, consider the costs, benefits and risks of applying it to different parts of the code. Consideration should be given to testing both the user interface and the API with automated test cases to check the consistency of the results.
Test Automation Framework (TAF)
A test automation framework (TAF) that is easy to use, well documented and maintainable, supports a consistent approach to automating tests. In order to establish an easy to use and maintainable TAF, the following must be done:
- Implement reporting facilities: The test reports should provide information (pass/fail/error/not run/aborted, statistical, etc.) about the quality of the SUT. Reporting should provide the information for the involved testers, test managers, developers, project managers and other stakeholders to obtain an overview of the quality.
- Enable easy troubleshooting: In addition to the test execution and logging, the TAF has to provide an easy way to troubleshoot failing tests. The test can fail due to:
- failures found in the SUT
- failures found in the TAS
- problem with the tests themselves or the test environment.
- Address the test environment appropriately: Test tools are dependent upon consistency in the test environment. Having a dedicated test environment is necessary in automated testing. If there is no control of the test environment and test data, the setup for tests may not meet the requirements for test execution and it is likely to produce false execution results.
- Document the automated test cases: The goals for test automation have to be clear, e.g., which parts of application are to be tested, to what degree, and which attributes are to be tested (functional and non-functional). This must be clearly described and documented.
- Trace the automated test: TAF shall support tracing for the test automation engineer to trace individual steps to test cases.
- Enable easy maintenance: Ideally, the automated test cases should be easily maintained so that maintenance will not consume a significant part of the test automation effort. In addition, the maintenance effort needs to be in proportion to the scale of the changes made to the SUT. To do this, the cases must be easily analyzable, changeable and expandable. Furthermore, automated testware reuse should be high to minimize the number of items requiring changes.
- Keep the automated tests up-to-date: when new or changed requirements cause tests or entire test suites to fail, do not disable the failed tests – fix them.
- Plan for deployment: Make sure that test scripts can be easily deployed, changed and redeployed.
- Retire tests as needed: Make sure that automated test scripts can be easily retired if they are no longer useful or necessary.
- Monitor and restore the SUT: In real practice, to continuously run a test case or set of test cases, the SUT must be monitored continuously. If the SUT encounters a fatal error (such as a crash), the TAF must have the capability to recover, skip the current case, and resume testing with the next case.
Things to avoid
With these maintenance considerations in mind, in addition to the important items that should be done, there are a few that should not be done, as follows:
- Do not create code that is sensitive to the interface (i.e., it would be affected by changes in the graphical interface or in non-essential parts of the API).
- Do not create test automation that is sensitive to data changes or has a high dependency on particular data values (e.g., test input depending on other test outputs).
- Do not create an automation environment that is sensitive to the context (e.g., operating system date and time, operating system localization parameters or the contents of another application). In this case, it is better to use test stubs as necessary so the environment can be controlled. The more success factors that are met, the more likely the test automation project will succeed. Not all factors are required, and in practice rarely are all factors met. Before starting the test automation project, it is important to analyze the chance of success for the project by considering the factors in place and the factors missing keeping risks of the chosen approach in mind as well as project context. Once the TAA is in place, it is important to investigate which items are missing or still need work.
Advanced Level Syllabus – Test Automation Engineer International Software Testing Qualifications Board. ↩︎