What does it take to test in DevOps? Key challenges and solutions

Jasmine Chokshi
February 11, 2021

Traditionally, testing has been perceived as a bottleneck in SDLC, something that causes delays in delivery. Organizations have long adopted the Agile/DevOps model, but not without its pitfalls and stumbling blocks, especially in achieving the ideal speed/quality balance. 

For enterprise DevOps, it is vital to rethink testing approaches to achieve agility at scale. There is much of an overlap of roles, for instance between business analysts and QA testers. Is the tester’s role diminished because of this overlap or because of automation? 

Since each organization is unique, with its own set of expectations and challenges in DevOps adoption, there is no cut and dried approach for testing in DevOps. Instead, there are some widely acknowledged best practices for continuous testing and achieving the best results using a multi-tool DevOps pipeline. 

Let’s take a deep dive into the reality of testing in DevOps:

QA Challenged

In the waterfall approach, the QA team gets a build that is deployed in the designated environment. Testers then start with functional and regression testing. The build remains with the team for a couple of days before the team signs off the build. 

The DevOps way

Testers now align their testing efforts to development teams in the CI/CD. The team ensures that all test cases that are fit for automation are automated and achieve optimum coverage. Environments are standardized and the deployment on QA boxes is automated.

In addition, all the pre-testing tasks, and post-testing tasks are also automated and aligned with the CI cycle. 

DevOps needs a high level of collaboration between the functions of the delivery chain. This blurs the boundaries between many of the roles of participants in this chain. Above all, everyone must contribute to the chain. So, a business analyst can contribute significantly to the QA process. Deployment engineers can add test cases to the repository, QA engineers can configure automation. 

Can you do away with the QA role?

While everyone can contribute towards building a better-quality product, nobody can replace the QA role entirely. There is a reason why QAs are vital to DevOps. Here’s a role that focuses on all the different testing strategies and overall test design. Having a dedicated QA engineer or team enables you to test comprehensively and cover all aspects of testing, be it manual, regression, functionalexploratory testing, performance testing, automated testing and so on.

What’s the ideal development/testing mix and balance for DevOps? 

Selecting the ideal dev/team testing approaches for DevOps

Since everyone in the chain is responsible for the quality and timeliness of deliverables, DevOps testing strategy will not work unless the full team is involved.

For example, to achieve automation at scale for each pull request and code change

  • Involve Business testers to perform manual and exploratory testing leveraging codeless test automation.

Include software developersas testing SDET to design functional and non-functional test automation scripts using a framework.

  • Developers can also create unit tests and build acceptance tests.
  • Quality Assurance and Dev need to collaborate and identify areas impacted by a particular build and execute the related test cases in addition to a sanity test pass.
  • It is important to configure code analysis and coverage toolsto ensure maximum code coverage.

Making the time for manual/exploratory testing

There is a certain expectation that DevOps enables deploying code weekly, daily or even several times a day. Where does manual or exploratory testing fit into this cadence?

It is ideal to test smaller bits of code changes and enabling the testers with tools to validate these on their own environments and utilizing feature flags. Ideally, manual testing should occur to some degree on each small code change. The best way to achieve this without impacting the timelines and agility is to identify targeted tests around what automated tests could possibly have missed. 

Exploratory testing can be undertaken on a pre-production level or integration level environment, again using feature flags to prevent breaking an experience on production. 

Using the right test management tools

Test management tools play a vital role in testing success by enabling DevOps teams to work in collaboration and speed up the release of high-quality software. The right tools offer superior integration with your project management, issue tracking and automation tools in the DevOps toolchain. 

The continuous nature of things in DevOps sometimes leads to alack of transparency and visibility. Using quality analytics, reporting gadgets and dashboards enables key team members and stakeholders with the ability to monitor progress closely even when test volume grows. Actionable insights and intelligence gained from this data help shorten the feedback loop and provide real-time visibility into quality. 

Everyone must adapt

The common perception is that in the new paradigm testers and QA must adapt. Just as quality must adapt from being a stand-alone function to an integral part of the entire development chain, so do the other functions. 

At the same time, testing in DevOps needs a more distributed, consultative model of testing expertise. This means the role of new QA is less transactional and more strategic, adding the most value where possible in the SDLC. Testers are now consultants for higher-level decisions regarding testing and quality. An ideal approach is to embed QA directly into the sprint or scrum team and creating a center of excellence or consulting group to advise on specific and strategic quality issues.

In the age of DevOps and Agile, you cannot approach testing in a conventional way. To scale and accelerate QA adequately for enterprise DevOps, businesses need to rethink their overall testing objectives and outcomes.

Let’s get you started with QMetry®

Get Started

Trusted by teams across the globe for 10+ years

The QMetry brand is trusted by 1000+ customers globally across Finance, Healthcare, Travel, Hospitality, Retail, Education and many more.
These are just some of our customers.