TEST AUTOMATION - WHAT to automate and WHAT NOT to automate

As we know, Test automation is one of the driving forces behind continuous delivery and agile practices. Many organizations wants to increase their pace of SW product deliveries, while they at the same time wants to maintain high quality. A significant amount of money are invested into Test Automation since fast feedback on software quality is an important enabler for shortening the development and release cycles. Test automation aims to increase the overall software efficiency and ensure robust software quality.

However, Test Automation is not an activity that replaces all of the manual work that is currently being done by the testers. The critical thinking performed by humans cannot be replaced by predefined sequences of clicks, keystrokes and API/web service calls; meaning not all activities can be automated. 

WHAT TO AUTOMATE?

Automation can be powerful means of enhancing testing efforts and making them more effective since it increases productivity, cuts cost and minimizes errors. But this can only become a reality once it is known what automation can and cannot do. Establishment of automated testing may fail if test automation is not applied in the right time, right context and with the correct approach. It is very important to determine what to automate, and more importantly, what not to automate to make sure that the automation provides value to the software development lifecycle. Only when we can identify where test automation fits and shall be used and when we know which activities that can preferably be automated, we can benefit from it and get the desired value. 

Deciding which tests to automate and which ones to leave for manual testing depends on a lot of factors, including but not limited to:

• The frequency with which the test is to be executed
• The effort needed to automate the test including upcoming maintenance
• The risk and damage associated with defects related to the test
• The stability of the software

Some tests are better suited for automation than others:

• Regression tests - these tests often cover basic use cases and critical business scenarios that tend to be pretty stable over time.
• API/Web service responses and database lockups that to not involve user interfaces are typically covered by test automation.
• Cross-browser and cross-device testing ensures that your application performs consistently on every single platform.
• Static & Repetitive tests - testing tasks that are used frequently and relatively unchanging from one test cycle to the next
• Data Driven Testing - application functions where the same functions needs to be validated with lots of different inputs & large data sets
• Load & Performance Testing - brings virtual users to generate a certain amount of load on the system

WHAT NOT TO AUTOMATE?

It is important to remember that automated testing only tests what the script says, so it will not go “exploring” for edge cases or abnormalities. Automation cannot and will not replace manual testers and not everything can or should be automated. But when is Manual Testing a better alternative than Automated Testing? Manual testing involves evaluation of the application under test by learning through exploration and experimentation and thus an assessment on the usability of the system. Manual testing is an investigation exercise which aim to obtain new information about the system that is not already known.

Some tests are better suited for manual testing than others:

• Subjective validation and critical thinking - application functions that must be validated subjectively such as usability and look-and-feel of the system (does the system provide intended value for end users?)
• New and changing functionality - application functions that are under development and are foreseen to be changed frequently and evolve over time
• Complex functionality - extremely complex functions may be a challenge for test automation (time and cost investment does not compensates the benefit)

CONCLUSION

When done right, Test Automation can definitely be a very effective means to improve the testing coverage and effort, but test automation alone will not be enough. We need to carefully select and decide which parts and activities that preferably can be scripted and added in to the Test Automation suite. Instead of seeing Test Automation as something that can replace testing, we need to see it for what it actually is; an activity that supports testing. By understanding how Test Automation can be used and make good judgement for when manual testing is the better choice, we can ensure efficiency and maximize the potential benefit of the test activities performed. So, rather than dismissing each other, automation and manual tests should be used in collaboration to get best out of both.