6 Questions to Define Test Automation Success

Jasmine Chokshi
January 17, 2019

What makes test automation successful? If you are driving the automated testing initiative at your organization, what exactly do you want from this?

To decide whether your test automation is successful or not, you must first set your objectives. Every company, project and team is different and at various stages of maturity in their DevOps journey.

What do you want automation to achieve for you? Take a look at some of these questions, and you will can determine your test automation criteria:

How frequently are your tests run?

The idea behind automating tests is to be get frequent test runs, often by default. So that you get the maximum benefit out of your efforts. If you haven’t set up your tests to run automatically with code changes, your automation will become stagnant.

If you make it a practice to run automation to run frequently, you will find the more troublesome aspects easily.

By testing both automation and system with scheduled runs, you can actually implement continuous integration and continuous delivery.

What is your manual dependency for test automation?

If you have to set up an elaborate manual process for test automation, you haven’t set it up correctly. Automation is by definition self-dependent. Ideally you shouldn’t devote many resources to this.

How much is your manual effort for automation? What steps do you need to take before you can view reports for coverage, execution, and other key details about the environment? Is there an elaborate process required before you run the tests?

If you seem to be spending a lot of effort manually on automating the process, you are not doing it right.

Do you execute a proof of concept( POC)?

One great way for getting it right is to implement a POC. It can provide far more information to you than the evaluation data, as long as your goals are clear.

Phrase the main criteria as the key question. For instance, can we create unit tests for in-tool custom scripts?

Can I replace my GUI client tool with Java abstraction layers and HTTP library?

You need to identify all the secondary goals as well as set the expectations and exceptions to this. Keeping POCs simple but answering the key, high-level questions will guide you better. Keep shorter time-fames for POC.

Can you trust re-runs?

Often when the test cases fail once, they are re-run and they pass eventually. In this scenario, what do you value? The first result or the second?

What’s the acceptance percentage for success in a rerun? Is a 70% success rate on reruns better than a 65%? Unfortunately, tests that sometimes fail are always not unreliable.

So tests that show up unexpected results even a fraction of the time are still untrustworthy. If you don’t know what caused that rogue result, then you can’t justify accepting the rerun success.

Can you execute tests in parallel to your code?

Test-driven development, behavior-driven development, and acceptance test-driven development are now fairly common practices. To be truly DevOps, you need to sync up your efforts, you need to execute test automation in parallel with test code.

There are many reasons why teams don’t go for parallel test execution. Sometimes your framework doesn’t support it, tests need to be sequential because they are not atomic, or they need hard coded values. By using a data driven testing approach that avoids hard coding values, you can increase maintainability of your tests and makes concurrent testing easy.

Design your automated test cases with parallelization in mind to avoid these hiccups.

What parameters have you decided for tool/framework selection?

Tool/Framework selection for automating your application is a critical success factor and varies for different types of applications under test. For instance, software/apps which interact with hardware/iot devices can not be automated through cloud service providers or normal record and playback tools where as a basic web application can be easily automated using free and open source tools as well.

The reason tool selection becomes more important is when application eventually grows, your tool should not stop you from automating because rewriting automation can be proved very costly.

Finally, automation efforts are by themselves as important and demanding as building commercially viable products. They deserve enough thought and assessment. Measuring everything you do and using the right metrics will help you navigate the course of automation correctly.

Automation testing is one of the best ways to shift left, deliver continuously and meet the complex testing goals of today. But choose your strategy and tool wisely. If you would like to find out more about QMetry Automation Studio, sign up for a free trial here.

Let’s get you started with QMetry®

Get Started

Trusted by teams across the globe for 10+ years

The QMetry brand is trusted by 1000+ customers globally across Finance, Healthcare, Travel, Hospitality, Retail, Education and many more.
These are just some of our customers.