| testing.com > Testing Craft > Techniques (Exploratory Testing) > A Survey > Workshop Report |
Bret Pettichord, 19 July 1999
The Seventh Los Altos Workshop on Software
Testing (LAWST VII).
July 10 and 11, Cupertino, California.
This was a two-day workshop on Exploratory Testing hosted by Cem Kaner and Brian Lawrence. The purpose of the LAWST workshops is to work through a particular aspect of software testing, analyze the state of current practice and try to reach agreements on principles or conditions that can be cited in our writings and teachings. The workshops are based on the assumption that teaching should be based on best practices found in industry.
Exploratory testing occurs when you dive in and start testing a product without working from a pre-defined set of test cases. Everyone at the workshop had experience doing this and agreed that it had value. Both Kaner and James Bach had already been teaching courses on exploratory testing and we began with them summarizing the material that they teach.
Kaner summarized the existing literature as treating exploratory testing as random or error guessing and in either case as something that was mysterious and difficult to rely on. He wanted to find more reliable principles to base it on. His presentation focussed on listing the various techniques used in exploratory testing. The perverse view is that exploratory testing is mischievous. He sees the task as finding ways for people to creatively design experiments.
Let me try to summarize some of the techniques that Kaner presented. One technique is to try to model the software by talking to the developers and creating an architectural model of the software. This can be used to facilitate group brainstorming and determining test strategies. Another is to act like a user and do scenario testing. One example of this is what Hans Buwalda (not a participant) calls "soap operas." These are tests where one assumes that the user has everything going on. For example, a plane reservation system could be tested by assuming the user had a honeymoon fare interrupted by a bereavement fare to another city that had to avoid an otherwise convenient transfer because of a murderous ex-spouse.
Invariance testing changes things that shouldn't affect the product and then makes sure that they indeed don't. Guerilla testing is a quick attempt to determine the level of risk of a subsystem. It can also be used to find major bugs early in the cycle. He cited that exploratory testing was a good technique for find reproducible cases of non-reproducible bugs. He characterized exploratory testing as generating bug reports and ideas and not generating archives or test cases. "What is the most promising line to find the next interesting defect?"
Kaner does not believe that there is a separate kind of person who is an exploratory tester. Ideally, he sees all testers spending 25% of their time doing exploratory testing, with 25% creating and documenting new test cases and 50% doing regression testing (i.e. running old tests).
Bach has applied cognitive psychology to software testing practice to come up with some of its elements. He characterized two paradigms of the professional tester. The clerical tester is focussed on creating documentation to define and justify the testing process (I later convinced Bach to re-label this the compliance-oriented tester). The cognitive tester uses an "interactive process of concurrent product exploration, test design and test execution". Although much of the software engineering literature is tilted towards the clerical approach, he found significant sympathy for the cognitive approach in the psychology literature. He mentioned that this corresponded to the difference between the analogic and analytic modes of reasoning. Analytic reasoning may be more respectable, but studies of experts have shown that they depend more on analogic reasoning.
He has been defining an exploratory testing methodology for use by Microsoft's Window 2000 Certification program. This essentially defines the process a tester goes through when doing exploratory testing and provides guidelines on how to document the process. This is a publicly available procedure that will soon be on their website. He has been training testers at Verisoft on how to follow it.
He characterized the elements of exploratory testing as configuring, operating, observing and evaluating. His methodology is based on what psychologists call a back-and-forth procedure rather than by a linear procedure. He sees documentation as not required, but as sometimes appropriate, as evidenced by the Windows 2000 procedure. He characterizes this as "making the world safe for insight." He also wants to provide the means for testers to fully justify what they did when they completed the task.
His test procedure includes the following tasks: identify sources of information, identify the intended purposes of the product, identify functions and data, identify oracles, identify potential instabilities, test each function and record failures, record smoke test guidelines and data.
He described several of the problems that testers have had with following this procedure. They had trouble listing functions and were very confused by his concept of an oracle (this is any means for determining whether results or output is correct). He also said that it was hard to get them to generate lists of questions and issues. They feared that this would make them look like whiners. In general, he concluded that exploratory testing was an application of critical thinking and that this might be difficult for some people.
Kaner and Bach gave their presentations with questions from the rest of the attendees. Questions were supposed to help them describe their ideas and were not supposed to be veiled criticisms of their ideas. That would be for later.
Several participants then presented what we call "war stories". These are stories of some use of exploratory testing on a real project. We use these stories to provide practical foundations for our conclusions. A couple of participants also shared short papers they had written on the topic.
"Non-exploratory testing can be like playing 20 questions when you have to specify all the questions in advance." -- Bach
We identified several questions to discuss and come up with some answers.
When is exploratory testing appropriate?
How can you supervise exploratory testers?
How well can exploratory testing scale in a large organization?
On the second day we reviewed our notes and ideas and tried to find some statements that we could all reach agreement on. Although considerable attention was put on how the statements were phrased, I didn't record them and am citing from memory.
We agreed on this definition of exploratory testing:
We agreed that exploratory testing was valuable.
We agreed that exploratory testing and predefined testing can complement each other.
We agreed that exploratory testing can be taught. Our challenge, then, is to find ways to do this.
Because of the format of the LAWST workshops, they are often characterized by debate, which can get acrimonious at times. Various adjustments had been made to the format to try to limit the amount of unproductive debate. It appeared that everyone agreed that this helped considerably. Indeed, this was the first LAWST workshop that actually completed ahead of schedule.
The workshop was facilitated by Brian Lawrence and III (pronounced "three). Participants were Cem Kaner, Noel Nyman, Elisabeth Hendrickson, Drew Pritzger, Dave Gelperin, Harry Robinson, Jeff Payne, Rodney Wilson, Doug Hoffman, James Bach, James Tierney, Melora Svoboda, Bob Johnson, Chris Agruss, Jim Bampos, Jack Falk, Hung Q Nguyen and myself. This includes 8 independent consultants, 4 heads of consulting companies, and 8 testers or test managers at software companies, including 3 from Microsoft. All have published writing or presented talks on software testing and quality assurance.
I expect to see several papers presenting the results of this workshop.
Be the first person to add a comment in the
Wiki Forum at page ExploratoryTesting.
(The Forum is explained in its FrontPage.)