Rethink Your Automation Setup
By Stratos - 8 min read
After spending countless hours working around framework issues, maintaining tests that are more hardcoded than the ten commandments, and investigating test failures that appear only during your absence, you realise something is wrong. UI automation shouldn’t have to be this hard to deal with or maintain.
Whether it’s unit tests, integration tests, or UI tests, there are plenty of posts and stories online that explain how those work. However, my team and I found there aren’t many stories which point out the challenges along the way and how to overcome them. Considering the amount of time we spent, I decided to document our story in an attempt to guide others facing the same problem and help them avoid pitfalls related to refactoring or retooling your automation.
What follows are recommendations and examples of how to approach the process of discovery, management buy-in, and developing better coding practices and frameworks for automation. These tips and suggestions might not work for everyone depending on what your development process is, but hopefully, it can give you or your team ideas and tips to begin your own process.
Refactoring More & Testing Less
The problems of maintenance, refactoring, and trying to work around an old framework, along with a number of other symptoms, are good indications that your test automation setup doesn’t suit your current product needs. Here are some examples you might use to determine if what you have currently isn’t working:
You spend more than 50% of your time maintaining tests You’re not sure how to structure new tests The developers are afraid to go anywhere near the test code The tests fail constantly with no code changes
An example of a messy test suite file from our previous setup:
Starting Over With A New Automation Process
Realising the issue is the easy part and the way to a new setup will likely not be paved with gold. As well as finding the right tools, concept and implementing it, depending on the company you are working in; you might have to persuade your team, boss or even the entire company about your intentions.
Step 1. Get Management Support
Test automation is an investment. Before you think about anything, you need to find out if the company is willing to spend time on test automation. There are plenty of case studies to back you up and tools to help you calculate the return on investment.
Not all products need test automation, more traditional companies with long release cycles might not require it. Even if you are working in a fast-paced agile team with multiple releases per week, you will be lucky if there is no resistance to change. Having exact figures, estimations and a solid plan are necessary to convince the right people.
Step 2. Choose Your Tools
Do the research. Make a matrix of the pros and cons of each framework and choose one, or a combination, based on your needs. Choosing the wrong tools will, at best, cost you a lot of time during prototyping and, at worst, leave you with an unusable test automation setup.
Things to consider:
Step 3. Prototype
Create the most basic setup and check that works for you. If you completed step two properly you will have proof that the setup works. A simple scenario will give you an idea of how the framework is performing and approximately how long your test suite will take to execute.
A successful prototype should be:
Selenium Easy have developed a simple prototype example you could use.
Step 4. Get Feedback
Ensure everyone is onboard with the selection of tools by showing off the working prototype to team members, other teams, and any interested stakeholders.
Explaining the reasons why you selected the specific tools, how you approached the prototype, and how to extend it, are important points in order to provide a full picture to your team. Getting feedback early will allow you to tailor the setup to the team’s needs.
Tormented by issues like unreliable tests, high maintenance costs, and time-consuming test authoring, our team decided to ditch the old test framework and start from scratch.
More specifically we decided to change the following:
Changing the tools ensured it was easier to maintain and write tests but there were also a number of flaky tests as a result of inconsistent coding. Therefore we decided to also proceed with a test refactoring.
Many of the test failures were the result of wrong test data used by the automated tests. Changing the data handling was a necessary step to clean up flaky tests.
Part of the test refactoring was also the division of the tests into smaller units in order to make them more manageable and reduce the overall running time.
We spend less time maintaining tests - The new setup allowed us to save a lot of time since we didn’t have that many flaky tests anymore. More specifically, we spend ~85% less time maintaining the tests.
The tests are now faster - Although the new framework is running on a real browser -making it slower- we can still run the entire smoke test suite in under 5 minutes compared to 25, as we are now running the tests in parallel.
Developing new tests is faster - Having better structured tests and re-using components allowed us to save ~50% of developing time.
Developers can now maintain the tests - Involving the developers in the process gave them the confidence to contribute to the testing efforts.
The tests are now accessible by anyone - Without the need for a local setup, anyone with the code can execute the tests in any browser, without the need of a local setup
Confidence In Test Coverage
Did all those steps make the tests 100% reliable? No, but it increased that percentage along with debugging and updating the tests much more easily. We don’t use the test automation to get rid of manual testing but as a way of early feedback and yet another safety net.
Regardless of the state of your test automation setup, UI tests are a fraction of the overall testing and if treated as panacea it can devour the team’s time without returning the investment.
On the other hand, if the right decisions are taken in all of the four steps, it can provide confidence, coverage and a great ally to your testing.
At the moment we’re looking to grow our dedicated QA-team. Want to be part of this awesomeness? Let us know!