The Need for Big and Public Examples of Software Testing

This is my editorial from Volume 1, Number 6 of Software Testing and Quality Engineering.
- Brian Marick

Suppose you’re a tester. You’ve just read an article describing "systolic testing," an interesting test design technique you’ve never seen before. You decide you’ll try it out someday. What happens?

• You don’t realize the technique applies in your next three projects, even though it could have. It’s hard to remember an idea you’ve never used.
• When you do try it, you run into problems. The bookkeeping that seemed so simple in the article’s example turns into a nightmare when applied to your huge system. You trust the author and believe that the technique can work, but he left out so many details that you need to know.
• You struggle with the "boundary" of the technique. In what situations does it apply best, and where would it make more sense to use other techniques? And how should systolic testing mesh with the other kinds of test design you do?

It’s all very frustrating. If only you could watch over an expert’s shoulder! But no one else in your company has even heard of systolic testing. You’re stuck.

Now, let’s suppose you’re Brian Marick, novice programmer in the early eighties. You’ve just read Artificial Intelligence Programming, by Charniak, Riesbeck, and McDermott, a compilation of design techniques for AI programs. You were struck by a technique that might be summarized as "solve a problem by implementing a little language that describes it well, then program in that language". Of course, the first three times that technique would apply, you forget about it. But, as it happens, you’re a user of the Emacs editor. You decide to write a "mode" in Emacs’s extension language to help you work with your company’s version control system. Along the way, you realize that you’re sort of working with two levels of "little language": the extension language itself, and the definitions that make up a mode. Later, you make some changes to the underlying Emacs source code, easily available over what wasn’t yet known as the Internet. You find more examples of this technique. And it keeps popping up in other source code you download and modify. The CMU Common Lisp implementation and the GNU C compiler have it. Oh, not in the fullblown AIP form, not by any means – but that’s OK, because the differences help you understand the boundaries of the idea, and how it meshes with techniques like data-driven programming.

This account is lightly fictionalized – nothing is ever as straightforward as that – but the fact remains that I learned software design techniques by reading and especially modifying big, complex, real systems written by some of the best programmers.

What equivalent opportunities do testers have? Precious few. Unless you’re lucky enough to work in one of the better projects in one of the better companies, you’re out of luck. And the sad fact is that the state of testing is bad enough that such projects are few and far between.

I aim to do something about this.

I am going to organize a public effort to test a real software product. Let’s suppose, as seems possible, it will be the AbiWord word processor (www.abiword.com). I’m going to start by pretending to be a tester assigned to that project. I’ll do and document exploratory testing, task planning, detailed test design, and my own style of test automation. I’ll try to recruit someone with experience in data-driven test automation to implement their style. I’ll do the same with someone who knows orthogonal array test design. Maybe there really is such a thing as systolic testing – I’ll recruit that person, too.

If you want to improve your testing skills, you’ll be able to examine what we leave behind as we perform real testing tasks. Or you can build on our work – nothing beats that for gaining real understanding. Perhaps you’ll apply orthogonal array testing to a different part of the product. Perhaps you’ll extend or modify the data-driven test framework. And then other people can learn from you. (I’ll help you with the documentation – I have, after all, just spent a year learning to be a good editor.)

I know this solution is incomplete. As Bob Glass points out in this issue’s Last Word, it’s wrong to assume that one particular project contains universal lessons. At least some of what’s done to a desktop application will be of no use to operating system testers. Still, a common point of reference has value. It’s easier to compare techniques when they’re applied to the same product.

I don’t expect people to flock to this project. I know that few testers test in their spare time, whereas compulsive programmers are everywhere. When I fiddled around with GNU Emacs, I got an improved tool that I used every day. I doubt anyone will get that from this project. However, I hope to persuade consultants like me that demonstrating their skills is cost-effective marketing. And I hope that non-consultants are motivated by the chance to parlay work on this project into jobs at those rare better companies.

Despite the difficulties, I want to try it. I’ve gotten a lot from the testing discipline over the years; it’s time to give something back.