Exploratory Testing is a software testing approach that is often defined as simultaneous learning, test design and execution. The person who coined this term in 1984 Cem Kaner defined it thus: “a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the quality of his/her work by treating test-related learning, test design, test execution and test result interpretation as mutually supportive activities that run in parallel throughout the project.”
The purpose of exploratory testing or ET as it is sometimes called is to approach testing intuitively. ET is widely used in Agile and DevOps models as it encourages discovery, investigation and learning. Above all, it is about the personal freedom and responsibility of the tester.
With test automation replacing many of the manual, laborious and repetitive tester tasks, approaches like exploratory testing allow testers to discover problems and edge cases that may otherwise go undetected.
Find out more about QMetry’s Exploratory Testing Tool
The need for Exploratory Testing
In the DevOps/Agile era teams are increasingly adopting continuous integration and delivering on the constant demand of quality digital experiences. Customers demand faster, better and more cost-effective products as their expectations have been set by a digital-first economy.
Speed is often the primary criteria in delivering these experiences. But haven’t we also heard of million-dollar bugs or UX disasters that cost dearly to the companies. The rush to deliver on deadline and lapses in testing have led to major financial and reputational disasters.
One reason for this is that most quality assurance approaches use a structured pattern. Test cases are based on pre-decided user stories. Similarly, test data is also structured based on the test cases.
Subsequently, teams measure test coverage using software engineering metrics. In most cases this may suffice and the coverage is enough technically. Yet, what this approach misses out on are the edge cases that are only discovered using UAT conducted with user personas in mind.
This is where ET comes in. With exploratory testing testers have the freedom to experiment with a user story and follow a certain sequence or pattern. As they keep noticing bugs, they can document them at the same time. Thus, they can convert a user story into a test case and use this information to dig deeper for QA.
Here the tester moves into test execution without officially authoring test steps. The exploratory tool and documentation are then the precursor to automation.
By using an exploratory testing tool, that allows you to automate the documentation, note taking etc. one can formalize the findings easily.
Scientific thinking in real time
James Bach, co-founder of the context driven school of software testing defines ET as ‘scientific thinking in real time’. While Exploratory Testing has always been around, it was referred to as ‘ad-hoc’ testing. However, this is a misnomer and the term Exploratory Testing was first introduced by Cem Kaner in the book Testing Computer Software.
Differences between Scripted and Exploratory Testing
So, now that we have established that exploratory testing is somewhat different from scripted testing, let’s look at the key differences between the two.
|Directed from requirements||Directed from exploration and requirements|
|Predetermined test cases||Simultaneous writing of test cases while exploring|
|Confirmation or validation of testing with requirements||Investigation and discovery of an application|
|Emphasis on prediction and decision making||Emphasis on adaptability and learning|
|Like reading from a script||Like making a spontaneous conversation|
Free Trial of QTM4J with inbuilt exploratory testing
The Exploratory Testing Process
A typical but not necessary the only approach to exploratory testing uses Session Based Test Management (SBTM) as follows:
- Classify common types of defects found in the past
- Analyse the root cause analysis (RCA) of these faults
- Identify the risks and formulate ideas to test the application.
- Test Charter
- Exploratory testing begins with test ideas
- The test charter helps in understanding how end-users will use the system
- Includes information like what to test, how it can be tested, what needs looking at.
- Time box
- Using time box method, a pair of testers work together for an uninterrupted, stipulated time period that is not less than 90 minutes.
- Time box sessions encourage testers to react on the system response and prepare for the right outcome
- Reviewing results
- Analysis of the defects
- Lessons learnt
- Analysis of coverage areas
- Reporting and evaluation of the results
- Comparison of the output with the charter
- Analyse if additional testing is required
The following execution details are important to the success of Exploratory Testing
- Identify or set out a clear mission or purpose for testing
- Create notes on what you are testing, why you are testing it and the assessment of the quality
- Document and track questions and issues that are brought up during exploratory testing
- If possible, pair up testers for better results
- The more you explore, the higher your chances of executing the right cases for the required scenarios.
- Documenting test coverage, risks, test execution log and issues or queries is equally important.
When to use exploratory testing?
- Exploratory testing is more ideal for some scenarios and not so great for others. Here are some scenarios where it should and must be used:
- If and when the team has experienced testers who can add more value to the project by doing ET
- When early iteration and feedback is required on a given product
- In case of a critical application where you don’t want to leave out any possibilities of an error or ‘edge cases’
- When you can’t decide what the next tests should be
- When there are new testers that have entered the project so as to check their work quickly
- To diversify the testing process after completing scripted testing
Exploratory Testing Examples
A meal delivery service website has the following components
- Coupon codes
- Top Rated
- Restaurants Near Me
A good idea to start the exploratory testing process would be to start from Login and then move on to restaurant choices.
Think from the end user perspective. Should login be the first step or only when the user has built a cart? How easy and intuitive is it for the user to order food? Remember, speed is an important factor in these types of websites.
Let’s assume you are working on a tricky project. Tricky in the sense that the requirements keep changing frequently. The entire logic of the application is changed and this impacts the flow. For instance, there is a flag set in a table of the product database according to the changed requirements. The flag must be removed from the database and its value needs to be added to the configuration file.
Since the requirements are urgent and frequently changing, the testers may not have tested all the configuration setting options. This is where exploratory testing is required. Once you start digging in and explore the depths of the configuration, you will find defects. It may require some exploration but you will ultimately find the deadly defects.
Pros and Cons of Exploratory Testing
Exploratory Testing promotes real-time thinking and helps in uncovering important defects that may not be easily found in the scope of scripted testing. It promotes use-case and scenario-based testing and uses more of an intuitive approach with less focus on formal documentation.
The emphasis here is more on learning and enhancing tester creativity. It is also useful to audit other testers’ work.
However, the success of quality of ET largely depends on the tester’s experience, skillset and expertise. Testers also need to be familiar with the application to conduct exploratory testing. It is not suitable for projects that have longer execution time.
Tips for Exploratory Testing
Use heuristics to guide your exploratory testing efforts. Remember, that test case execution and test case creation go hand-in-hand. Be prepared to take notes and record information when you explore. It is also helpful if you can use an exploratory testing tool that automates this process for you.
The test cases may keep on evolving based on tester observation and learning. Testers can use various testing techniques such as boundary value analysis, equivalence testing. To make the sessions more structured and focus, use session-based exploratory testing.
Importance of Exploratory Testing for Agile or CI/CD approach
When you automate the most common workflows aided by conventional scripted testing, it frees up manual testers to explore. It allows them to quickly identify the most important defects and help the application meet the needs of end users. One significant advantage of exploratory testing is finding major real-world bugs faster and this fits into the DevOps approach.
It helps testers match pace with the speed of agile workflows and values testers as core part of the development process, sharing the values of the agile manifesto. As development sprints keep shortening, there is a greater risk involved in missing out on some crucial bugs. Exploratory testing can help you focus on the high-risk areas to find mission-critical defects.
Even while doing unit testing, you can use aid your efforts with exploratory testing, document the steps and use the information to test extensively for later sprints.
Tools like QMetry Test Management for Jira with deep Jira integration are useful for Agile teams with their in-built exploratory testing feature. Testers can prioritize feedback and implement it quickly without the need for additional tools.
In the rush to deliver faster and better software there is always the unnamed risk of missing a critical defect. Exploratory testing should be an integral part of all software testing plans and approaches as it helps cover more ground, faster and relies on tester’s intuition and experience. Something automated tests can’t provide.
Try our Exploratory Testing Tool with QTM4J
What are the challenges of Exploratory Testing?
The benefits of Exploratory Testing are largely qualitative so quantifying the gains and setting the right expectations can be difficult. One more challenge is the ability to log defects and document them with proper evidence as the tester continues to explore undocumented scenarios. But you can always use an exploratory testing tool to make this process automated.
Exploratory testing demands a certain level of skill and experience and not all testers will test equally.
What is Exploratory Testing in Agile?
Exploratory Testing shares some important values with the Agile manifesto. The high frequency of release in the Agile method requires testing to respond to feedback quickly and easily. Exploratory testing is both quick to execute and flexible for adaptation. It can help teams find bugs that automated tests miss and allows a fresh perspective to discover more bugs. One key reason why ET is effective in Agile is because of the speed of its cycles. Unlike other testing approaches, there is no need to invest time in authoring, preparing and approving test cases where exploratory testing is concerned. It reduces unnecessary documentation which is in line with the values of Agile Manifesto ‘working software over comprehensive documentation’.
What is Exploratory Testing in Software Testing?
Exploratory testing is simply a hands-on approach of testing where testers invest less time in planning and formal test authoring and more time in learning about the application from the functionality standpoint. The test plan involves creation of a charter and a mission for the short and time-boxed test effort.
Why do we do exploratory testing?
Exploratory Testing is important because the people who design and develop software have preconceived or cognitive biases about the function of the software. It is all the more important as software complexity has increased. Software covered by automated tests may regress even after bugs are fixed. Exploratory testing allows us to take an impartial view and test the software as a whole without looking into the individual elements that make it.
What technique is used in exploratory testing?
Proponents of Exploratory Testing recommend approaching it like a Test Tour. There are many techniques that come in handy such as equivalence partitioning, error guessing, decision table testing, boundary value analysis etc.
What is the difference between ad hoc and exploratory testing?
Exploratory and ad-hoc testing were used interchangeably sometimes in the past. However, there is a difference. Ad-hoc testing requires learning the application first and then working with the actual testing process while ET has no such mandate. You can start exploratory testing with a basic context of the application and learn on the go. Documentation is necessary for Exploratory testing and it is necessary to document the details of testing activity. This is not a basic requirement for ad-hoc testing.
Are there tool for Exploratory Testing?
QMetry has been one of the early and most vocal advocates of using exploratory testing. As a result, we provide a built-in exploratory testing tool. With our exploratory testing tool, testers can record test sessions, take screenshots, annotate them, record voice memos and make it extremely easy to share the observations with the development team. One of the most important features of QMetry’s Exploratory testing add-on is that users can create test cases from sessions recorded during ET. No need to write test cases manually.
Is it possible to use Exploratory Testing Tool in Jira?
QMetry’s Exploratory Testing extension works inside Jira providing deep integration. All documentation and notes that you generate can be easily linked to Jira issues. The resultant bug reports can be escalated quickly and therefore teams save time. Testers can also log defects to the scenario while recording the test session and the defects will get logged in Jira. You can even log issues without recording a session by adding them as bugs in QTM4J.