Nicky Tests Software: Introducing people to Exploratory Testing Part I

Thursday, October 12, 2017

Introducing people to Exploratory Testing Part I

 A bit of context

For the past 2 months(ish) I've been working on introducing Exploratory Testing to people in my project, starting with people in my immediate team of 3 testers, which is distributed between 3 countries. The project, as a whole, has a lot more than that, but the plan is to introduce  this as a pilot, see what the testers think of it, and then (hopefully) introduce this approach to other teams and other features.

I'm still a relatively new person on this project as I've been on it for 6 months, but the other testers in my team have been on this project for 3-5 years. So I've made sure to ask their thoughts, listen to their ideas and address their concerns about this - they know things about our context (which I don't) because of their experience here.

Currently, on the project, we write test scripts, link them to test cases, then execute those test cases. I'm under the impression that a lot of people on the project have only ever used test cases to formally do testing (when they're not doing "Exploratory Testing").

Another thing to keep in mind, is that this process is still ongoing (hence 'Part I'), but while these thoughts are still fresh on my mind, I wanted to get them down.

Sites/resources I shared

Spotify Offline: Exploratory Testing by Rikard Edgren
I asked my Test Manager (who also thinks Exploratory Testing is effective) about resources I can share and she recommended two Youtube videos by Rikard Edgren.

James Bach has a lot of useful posts on his blog helping explain what Exploratory Testing is.
Few examples:
Exploratory Testing 3.0
What is Exploratory Testing

I also shared a post from Michael Bolton's series - what Exploratory Testing is not

Lastly, I decided that Session Based Test Management (SBTM) would be a great way to help us structure our Exploratory Testing, so I shared some resources around that including this powerpoint by Anders Claesson

Managing others' expectations

I've noticed that a lot of people on this project have a very different understanding (to me) of what Exploratory Testing is. Based on what people on this project say, it seems that they think Exploratory Testing and Ad hoc testing are the same thing.

Since initially introducing the idea to the other two testers I work with, I'd say I've been met with cautious reception. Test cases is seen, by one, as proper testing and Exploratory Testing is not-  I'm still working on breaking that misconception. Aside from that, it does seem to be a welcome idea - you get to see results faster and are able to react to what you find as you test.

In terms of time estimates and how this affects our team meeting it's goals - I've been sure to communicate with our team that this is a new way of working which we need to learn - so any time savings may not be seen straight away.

Lastly, there is the idea of coverage - to cover this, I've decided to specifically mention, at the start of each charter, which Acceptance Criteria is covered in the charter. People on this project like reports and seeing the number of passed test cases etc. - I'm still learning how to deal with that mindset and any obstacles which arise there.

Managing my own expectations

This has been tough. It's been a while since I've been in a work environment with people avid fans of test cases. I don't think that test cases offer no value at all, but I think people overestimate the value it provides.  Test cases can give people a false sense of security of the state of the product when they see that 95% of their test cases have passed, but they can't properly attach meaning to how a 95% pass rate affects the customer. It's just a number.

I've also been trying to manage my expectations around other people's understanding of good testing and my own understanding of good testing. I constantly remind myself that their understanding is based on their own experiences. So neither of us are necessarily wrong - we are both right in our own mind.

My goal is to effectively show people on my project another way of doing testing. Then they can make more informed decisions in the future and choose which approach is best for their context.

Moving Forward

I'm hoping to sit with the tester who'll soon be on-site and pair test with them as we do Exploratory Testing. I should also organise a pair testing session with the tester who'll still be offsite. Once we've done this, I'll seek more feedback on this approach and see what they like and what they are concerned about (in the context of this project). We're also working on a Low Tech Dashboard to help us communicate the testing status for our features and help others attach meaning to what we present. 

In time, I'm hoping to help introduce this approach to other teams in our project - but for now we need to continue the pilot first.