Leaders Insight into Building or Buying Test Automation Frameworks for enabling high software quality delivery

Why Test Automation ?

1. Test Automation on mind - So your enterprise plans to invest into new test automation framework for API, UI and Data Related Testing for shifting quality left (early feedback)
2. Enough has been said about "why do test automation" and I will leave it with readers perspective. It never needs florid exaggeration.
3. What matters to your business - Improve Software Quality.
4. Time is essence-ROI and allowing one the ability to see sustainable long term growth.

Before jumping on the bandwagon to get the Pink shiny unicorn testing framework, consider following-

• Test Phases - Typical Test Phases alleviating quality are as following (disclaimer -- There are various definitions of each of these testing types and I will not get into their definitions, I will let the reader skin the hair strand). Focus on right test phases to be automated.

Unit Testing

API Testing

Integration Testing

System Testing

Functional Testing

System Integration Testing

End to End testing

User Acceptance Testing

• Multifaceted Framework - With the exception of Unit Testing, including some more specialized testing types like ETL, Performance & Security testing etc, typical test phases can be potentially targeted to be automated using a single multifaceted test automation framework. In this article a general reference to testing framework would mean more like the one's mentioned in Gartner's Magic Quadrants including open source (Selenium,Robo etc) , but I will specifically exclude referring to "unit testing & other specialized testing types" in this article, unless it is mentioned explicitly.
• Technical Criteria - This is an easier one to knock off as the choices are binary or finite. Ask "Does the framework meet the enterprise wide technical needs ?" More troubling question here would be "Should it meet enterprise-wide needs and why ?". More tools would mean more operational expense.
• Time for a cliche - "Not everything can be automated". Look for a balance between faster " time to value" Over super-swiss army knife of testing framework which accommodates all the nuances of the all the test types and its users in the world.
• Ease of use- When it comes to building quality software, you would want to spend time on building software and have ability to build high quality tests with minimal effort (& investment) into it. Consider that if you expect Business Users to end users of the framework (Hint-They should be) and not just the dev team, then can they get on with the framework with ease?
• Shelf life- Testing Frameworks have a shelf life, yes you heard it right, & its finite. A time will come when the framework at hand becomes obsolete and then comes teething migration to newer framework. Does the framework have it more like 3-5 years ?
• Maintainability- There is nothing like "Free lunch", open source software's have low entry barrier (cost & learning both) but are plagued with need to maintain it more than often.
• Rapid Collaboration- This is the elephant in the room. Believe me. Reusing testing component, rapidly enables the team to do more testing than building one. Why limit the collaboration to a team or teams or department , why not enterprise ? even better how about complete organization ? One of the way to think about it is that test components should be pushed automatically to testers when they available and not expect the testers to download one when its available. Finding test building components for reuse is quite easy only when testers has been provided with the ability and habit to find one.

What to expect from a testing framework ?

1. Support Parallel Test Execution, this will alleviate shifting quality left. A big time.
2. Dynamic Execution infrastructure - Ability to spin up and down the Infrastructure resources used for firing tests will optimize resource utilization (and in-turn costs)
3. Extensible beyond tests and dev folks- It should be fairly easy for someone without "test automation" as primary skill to justly use framework with minimal training.
4. Multi-Language Script- Allow the test components to be written in various languages but within same framework (Python & Java are favorites), this should give a leverage to the teams to hire extended team members without too much technology constraints.
5. Reuse of Testing components - The framework should allow reuse of most of the test scripts building components across the enterprise and not just the team or few teams. Watch out for the "Fallacy of Reuse because we have SCM". The framework should facilitate explicit components sharing across enterprise. Theoretically sharing should be possible by simply using the SCM, unfortunately most prevalent practice is to write own code over using someone else's test code.
6. Extensible to various testing types- It should support UI based testing, API Testing, Data related etc.
7. Compatibility with Enterprise tooling ecosystem- The framework should ergonomically fit into the existing tools ecosystem. These days CI tools are at the the center of all the changes in the ecosystem, test framework should be able to integrate with CI tools with ease. Web-hooks are also amazing when it comes to running the test suites from Slack or other persistent chat/collaboration platforms.

What the heck are you against - Advantages Vs Challenges

• Open Source

There are lot of open source frameworks available, feel free to use them. But remember, frameworks can be confusing terminology for leaders. For example lot of you may say "we have a Selenium based framework". Is that really the case or did you build one around it ? If later, then one is asking for a significant investment in building one. However, if you get your hands on "ready to go" open source testing frameworks then this will potentially decrease your "time to market".

• Commercial Testing Tools

With every extra dollar spent, one may get better service offerings from the tools OEM. However, OpenSource frameworks still give these proprietary tools a run for their money. Commercial tools provide heightened security (or sense of security, a debatable topic for later time), many of these support extensible use of AI to reduce the test flakiness & automated scenario creation. Such tools also tend to provide less flaky testing experience which may be good thing if you think that's important for your business (a flaky experience is very bad thing). Generally these tools cost anywhere between $300 to 8000$ per user license each year, with median feature tool inching towards a price point of ~$2500.

• Homegrown

This is a very easy choice, most popular choice when used in conjunction with open source testing libraries/frameworks. By the way it is also a favorite opinion from most testing teams. Such tools are aren't exactly free as it mandates the need of effort to be invested in building a wrapper framework. That means one has to incur upfront price before test zero is created for fair usage. But the advantage is that such frameworks can be modeled to suit enterprise wide testing nuances (the good, the bad and the ugly). This may or may not be necessarily a good or bad thing. As customization's evolve over a period of year or two, these frameworks pose a challenge with a need to support such change and people just roll over. So one would argue that isn't that's how software's work ? Absolutely yes, but is that part of core business delivering quality ? Or do you want to invest your effort in another piece of software for internal consumption in presence of opensource options ? If yes, then think about how can this effort be reduced or eliminated ?

• Lifespan of an Testing framework

Or software shelf-life as we may like to call it. How long would a testing tool be leveraged in an enterprise ? This is something which should be considered before choice is made to buy or build. I believe in the thumb rule that it will last not more than 4 years before test migrations hit with the advent of new tools. In such case, high warm-up during initial times combined with effort investment customization may negatively impact the ROI.

Metrics

Spoiler alert "I am writing about metrics (in short) which will in-directly align with the definitions of business value of Agile community, direct co-relation will lead to this article swaying away from the core topic of this article". Pardon me.

Pass & Fail figures are good and emailing them is BAU, but reports generated from each test run would yield much more value when these are stored for trending purpose. Also co-relating the test run outcomes with defect detection trends and delivery cycle times will pave the way for further quality improvement. Consider results integrations with popular analytics platforms (Loki & Grafana, ELK stack etc).

By the way, "Test Re-usability Index" is an interesting metric to have. This will allow leaders to gauge reuse of testing components.

Closing Lines

Whilst UI heavy automation frameworks have been extremely popular but Unit & API testings have been able to deliver quality faster than ever. The UI heavy automation frameworks provide finite ability to shifting quality left. The speed and accuracy of Unit Testing and API testing are well beyond such UI based testing and these should be super preferred over UI based frameworks. For more reading on inverted testing pyramid, see this blog on Google. The essence of this testing pyramid is more testing needs to be done in unit testing than any other phases.

Author:

Mahesh Raut, Leading DevOps Transformation

You can reach me on maheshpraut@gmail.com