Exploratory Testing

What is exploratory testing?

There is always more that can be tested than you have time for. A tester’s mission is to best choose where and how to spend their time.

  • wandering - purpose = lost
  • wandering + purpose = exploring
  • exploring + judgment = exploratory testing

Exploratory testing allows the tester to balance risk (the consequences of a failure) and coverage (observing all possible behaviors). It brings test design and execution together: you use the information you’ve gathered so far to immediately change what you’re going to do next.

How does exploratory testing fit in with automated tests?

The code in automated tests tell you what’s expected (at the time the test was written, by the person who wrote it). The output from the automated tests tells you what you got (at the time the test was run, to the person paying attention to the output).

Automated tests can’t do the evaluation work to tell you if the difference between what you expected and what you got is risky. They can’t answer questions like:

  • Did we build the right thing?
  • Has what we expected changed?
  • How does it all fit together?
  • If what we got has changed, is that a problem for the customer?
  • What didn’t we think of?

Valuable testing includes both automated and exploratory testing.

How do you do exploratory testing?

Testers keep these things in mind as they’re exploring an application. (Skilled exploratory testers can describe their thinking to their team later, or even as they’re doing it.)

1. A basis for comparison

When you find something you don’t expect, you’ll need a way to explain to other people why you don’t expect it. In deciding whether something is unexpected, you might find yourself referring to:

  • the history of the product
  • the image of the product and organization
  • claims made about the product by marketing, sales, documentation, conversations, user stories, etc.
  • user’s expectations
  • other behavior within the product itself
  • the product’s purpose
  • statutes and standards (legal requirements, SLAs, accessibility standards)

2. Rules of thumb and checklists

Having a list of ideas of things have gone wrong on software in general can help you identify similar patterns in your own product. They won’t prevent the unexpected, but having some ideas at your fingertips may help you uncover unexpected things earlier.

3. Deciding what to focus on

Setting a mission for your exploratory testing helps you decide what’s in and out of scope, and what you’re trying to accomplish so you don’t get lost. (Exploring is wandering with a purpose.) Try writing down:

  1. where you’re exploring (the test environment, the new feature, etc.)
  2. what you’re using/how you’re exploring (the automated tests, the logs, the accessibility scanning browser extension, etc.)
  3. what question you want to answer (are the existing tests passing, are we logging at the right level, has the extension uncovered new issues, etc.)

Some examples of directions for your mission:

What’s hard about exploratory testing?

It’s not just poking around! It can be hard to describe why, when, and how to do it. Keeping all these things I’ve listed in mind all at the same time takes practice. It’s hard to know when you’re done, or if you’ve done enough. And it’s usually best to do both exploring and automating, so finding time can be tricky or hard to advocate for. Having the brainpower to be actively learning the whole time you’re doing your work is hard.

All the links in one place