Thoughts about Automation Frameworks

In this article I’ll be talking about my experience with automation frameworks. But this time I have more questions than answers or suggestions.

I have been working in Testing and Automation for all of my career. I have worked in a company which provided Product development services to other product companies as well as worked for a product company. I have tested enterprise software applications as well as software used on hardware devices (thin clients). Some of them were CLI based, whereas some had both UI as well as the backend to be tested. I was fortunate enough to get an opportunity to design and develop automation frameworks for some of them.

The first time I did write a framework, I didn't even know how a framework is designed. I just went ahead and started developing it. It was a CLI for an ETL software, the automation was to be done in Perl as the earlier framework was also developed in Perl and I had to learn the language as well as develop the framework. In the end it turned out very well. 

Going ahead, I worked in QTP for a web-based UI application, that did not turn out very well. The customer whose software we were testing, was also involved in the automation, and somehow the framework became very complicated. Some parts of the framework or rather code had become so complex, that I stopped trying to understand it and just used it wherever it was required. I admired the person who wrote it as the code accomplished many things which we thought could not be done, but became aware how difficult it is to understand someone else’s code. I also realized that I was able to comfortably fix any issues in the automation code rather than write such complex code myself.

Further ahead I tested software which were very difficult to automate and the customer themselves did not expect any automation to be done. Some of them involved hardware devices, some involved combinations of physical and virtual machines, and some had telecom equipment or simulators for them.

Next started my Selenium and Python journey. By this time I had a fair idea of how frameworks should be and started working on it. We also started using Python’s unittest for API automation. On the UI side, we had started with Selenium RC and moved to WebDriver later. We continuously tried to improve our Selenium based framework and then came to a point where again it had become too complicated in some areas and flaky during the execution runs. We then graduated to using BDD based Robot Framework which again used Selenium libraries. We were happy with it, it turned out very well. We even integrated it with our API testing framework so that we could utilize it for cleanups, teardowns and actions which were not the target of our tests. But as the product evolved and became more complicated, so did the framework. Later we came across a lot of flakiness during the test executions. The UI and workflows were continuously evolving, tests had to be frequently fixed, and results were inconsistent both because of the product changes as well as timing issues with loading of the UI.

We had set up a nightly pipeline on Jenkins and divided the tests to run on multiple VMs so that overall execution time is reduced. However, we were overwhelmed with the amount of tests, number of failing tests to debug, VMs facing memory or connection issues causing tests to not run, product changes which required changes to existing automated tests, and new automated tests which had to be accommodated in this whole system. If some tests were not executed for a few days or weeks, we were sure that they would fail and need changes. Plus there were issues with the Robot Framework requiring specific versions of different python modules that it required. This was a long time back and earlier versions of Robot Framework, I hope they have improved upon the compatibility now. Also we had to continuously update the selenium drivers required by the browser as some actions in the tests would stop working in the latest browser version if the driver was also not updated or not yet available. We also had a case where we had to freeze the Firefox browser version to be tested against, as the corresponding updated driver did not have a fix for a very long time. There was a constant pressure from the management and questions as to why new tests were not being automated quickly or why the number of failures was not coming down. It became difficult to answer these questions. We were spending most of the time debugging and fixing, instead of automating new tests. 

Why did we land in such a situation? Was the framework so complicated that it became difficult to debug, fix or update the automated tests? Was it a problem with our infrastructure - slow VMs, less memory, unstable network? Was the product changing too frequently and hence was not a good target for UI automation? Should we have concentrated more on API automation than on UI automation? Was the thought process correct that we should automate the use case which our customer would exercise hence we should pay more attention to UI automation? Was the management correct in the assessment that we were not productive enough in terms of automation? Was our choice of automation framework and script language (Robot, Selenium, Python) right? Were we not using the framework in the correct way? Were we spending a lot of time and effort on the automation framework than on testing the quality of the product? A lot of questions with no specific answers.

In the next project, as a change we explored a few other tools and zeroed down on Cypress which is one of a very few tools that is not based on Selenium. Our main concern was flakiness that we suffered earlier with the Robot Framework. Cypress turned out better than Robot initially. However we faced new issues here. Cypress did not have cross domain support and it did not support testing in multiple browser tabs. A lot of our tests remained non-automated because of these shortcomings. Earlier we had thought of using some workarounds to overcome these cases but it did not work. Also, Cypress required JavaScript skills which increased the learning curve. So was our choice of framework incorrect? We then thought of using another framework to automate the tests that were not possible using Cypress. We came down to two choices - Robot framework or Pytest with Selenium. We heard that Robot framework had improved a lot lately however using Pytest would enable us to integrate with our API framework easily as that too was developed using Pytest. But is it a good idea to have two different automation frameworks at the same time? Should we migrate the tests to the new framework or continue to maintain both? Should we follow a philosophy of using whatever way to automate the tests even if it results in multiple tools and frameworks? Should we put more efforts on API automation than on UI automation?

Sometimes it feels like a lot of unanswered questions and sometimes it feels like multiple answers for the same question. However many times, it is the quality of the product that suffers because we have to put undue efforts in solving the automation issues rather than testing the product. Or sometimes we spend very less time in solving the automation issues and then think why we are not getting enough output from the automation. Where should we draw the line and decide that automation is not helping us and something needs to change? I hope we get the answers and people understand that automation is not a magic wand that could do all the testing easily. Automation is just a means of executing the tests, and requires efforts to develop and maintain.