Last month Gil Broza kicked off his “Spot On Interviews” by interviewing Michael Bolton. Gil and Michael discussed “Is There A Problem Here?” including:

• Testing as an Exploratory/Investigative vs. Confirmatory Pursuit
• The Trouble with the Ubiquitous Testing Goal of Completing Tests
• Are you Testing, or are you Checking?
• You Cannot Test Everything, Where do you Invest Your Time?
• Agile Testing

I enjoyed the recording immensely and made several pages of notes. Although many of the discussion points shared insights I was already familiar with I appreciated the reminders. I found Michael’s perspective and approach to articulating the challenges in testing to be very helpful. I have captured a variety of the discussion points to share with you.

Exploratory vs Confirmatory Testing

• Exploratory testing is a process where we are seeking new information.
• Confirmatory testing is a process where we are verifying and validating things that we thought or hope are correct.
• Testing is a thinking process and we need to be using critical thinking skills. “Critical Thinking is thinking about thinking with the intention of being fooled.” Michael encouraged testers to be mindful of our capacity to be fooled.
• “It Works.” According to Jerry Weinberg this is the shortest and most ambiguous sentence possible. I believe that is very true and yet it is used so often by testers when questioned about a feature or program they are testing.

Test Planning in Exploratory Testing

• Planning can be expensive. There are long feedback loops between planning to find something out and then finding it out.
• “With Exploratory Testing we want to try to find out, in as many ways as we can, as many things as we can, about the program.” Michael encouraged testers to consider that the traditional test planning approaches are not effective in accomplishing this as it leaves out a great deal of learning.

Testing vs Checking

• Testing vs Checking was covered in the interview. Check out Michael’s blog entry Testing vs Checking for full context on this excellent discussion point.
• Michael shared an excellent reference from the book Computer Programming Fundamentals by Gerald M. Weinberg and Herbert Leeds. Back in 1961 they wrote that testers are not really in the job of proving repeatability, machines can do that, we are really looking to prove or test adaptability by the machine. The machine can handle whatever we through at it. In order to do this well we need a suspicious nature and a lively imagination. I love that!

Practicing Exploratory Testing

• Indulge your curiosity. Follow where your research leads you. Be innovative.
• Play is exploratory learning. Michael referenced the book Play as Exploratory Learning: Studies of Curiosity Behavior by Mary Reilly. The book talks about how we have de-valued play. Kids learn faster than anyone; and kids in pursuit of knowledge are not even aware of their learning.
• Michael shared his excitement with the Weekend Testing Movement. This was started by four testers in Bangalore. Acting as their own managers, coaches and critics, they find a piece of open source software and test it. As Michael put it, every weekend these testers are sharpening their saws. You can read more about the Weekend Testers here http://weekendtesting.com.

Test Coverage & The Impossibility of Complete Testing

• “As a tester you are acting as the eyes, ears, nose and fingertips of the client. You are a sensory instrument for your client.” For me, this quote from Michael is one of the best definitions of a tester I have ever heard. Excellent!
• As that instrument the testers job is to identify the kind of information your client wants. There are things we know we don’t know, and then things we don’t know we don’t know. There are ways to find out about the things we don’t know we don’t know.
• Development and testing are constantly dancing with circles and feedback loops. It is impossible to schedule invention and discovery. I felt this was another excellent perspective.
• What is done? Do we honestly believe we know enough to move on? Have we achieved good enough? Can we take the next step with this product (may be a release, or next feature, etc.)? There is no “standard” we can reference to define this.
• One approach is to consider when the cost of the task exceeds value of the return.
• “Testing is always sampling strategy. The number of conditions we can test is infinite. The number of platforms and variations on platforms that we can test is intractable if it is not infinite. Perspectives that people have on value are infinite. The input domain of valid input for any program is potentially finite but certainly is intractably large for anything except the most trivial program and invalid inputs are infinite. No matter what we do, we are always doing some subset of the testing that we could do that is asymptotically close to zero.” This was a great description!
• Michael referenced “Try Again, Fail Again, Fail Better, Fail Faster” by Samuel Beckett.

Regression Testing

• Michael referenced the RCRCRC mnemonic for regression testing by Karen N. Johnson. The mnemonic stands for:
  • Recent: new features, new areas of code are more vulnerable
  • Core: essential functions must continue to work
  • Risk: some areas of an application pose more risk
  • Configuration sensitive: code that’s dependent on environment settings can be vulnerable
  • Repaired: bug fixes can introduce new issues
  • Chronic: some areas in an application may be perpetually sensitive to breaking

  Here is the link to Karen’s blog entry on this mnemonic: A Heuristic for Regression Testing.

If you have not already heard the interview, I really hope you will take the time to do so. There was a wealth of information covered and each topic warrants a deep dive to fully grasp the concepts presented. I believe that for each of the concepts Michael shared there is a extensive information available on the web in the form of blogs, articles and white papers. I encourage you to indulge your curiosity, follow where your research leads you, and to be innovative in applying your new learning.

Here is the link to the recording: Spot On Interview with Michael Bolton. Enjoy!

One Response to “Do You Indulge Your Curiosity?”

1. Lynn McKee says:

   May 17, 2010 at 5:13 pm

   As an FYI, I was not able to “quote” the reference from the book “Computer Programming Fundamentals” by Gerald M. Weinberg and Herbert Leeds as I do not have the book to get the verbiage from. Anyone have it and willing to look it up for me?

   I have a long list of books to buy this summer. Hoping to grab this and others written by Jerry at CAST where his publisher usually has a table set up.

   Cheers,
   Lynn