| testing.com > Testing Craft > Techniques (Test Support Code) > Case Study |
Created and summarized by Brian Marick
Summary
Much testing is indirect: we look for bugs in the way a program builds some internal data structure by examining the output produced from that data. It can be hard to find mistakes that way. Adding a smallish amount of test support code to the product can make testing more efficient and effective.
I show an example of how I did that. As is often the case, I was able to build on code the developer had already written, which considerably reduced the effort. In this example, I was the person who added the test support code to the product. Were I not a programmer, I would have needed to persuade a developer to do it. Since the amount of work was small, that would not have been too hard - provided the testing team had good relations with the programming team.
I also show a wrinkle on the practice of saving files of expected results. Instead of saving expected data, I save expected scripts.
Background
I'm a tester contributing to the AbiTest project, which is testing the AbiWord word processor. I decided to focus my effort on tables. At first glance (and maybe at second and third), that seems odd. At the time I started (November, 1999), AbiWord didn't support tables. None of the "upstream" work (requirements, user interface design, design of internals) had been done, and it wasn't likely to begin soon. However, a number of things about tables will someday make for interesting case studies, and there was one table-related testing task I could do. So I resolved to start with that.
AbiWord can import Word documents. It makes use of a library named wv. So there was table import code to test. But without AbiWord table support, how could I test whether table import was done correctly? The answer: wv is built into a program called wvHtml that converts Word documents to HTML. Here's a picture of the process:
I could test whether table import worked correctly by looking at the HTML output in a browser. Presumably, if the browser displays an incorrect table on the screen, wv's internal data structure was wrong.
What went wrong
That turned out to be completely naive, for several reasons:
For the gory details, look here. That set of pages is an embarrassing example of doing dumb things because of schedule pressure. I had announced these pages in an editorial in Software Testing and Quality Engineering Magazine. I wanted to make a substantial example available around the time subscribers received that issue, so I persisted in testing by looking at browser displays long after I should have known that was a bad idea. I didn't think I had the time to do what was obviously necessary: learn more about how Word, HTML, and wvHtml handle tables. A classic case of "never enough time to do it right, always enough time to do it over."
What makes sense
Rather than caring about the whole picture above, I really only care about the shaded part in this picture:
I care about whether the wv library, as used in wvHtml or elsewhere, creates the correct internal representation of Word tables. That's the data structure that AbiWord will eventually convert into its format for tables. My proper task is to find a way to examine that data structure in the most cost-effective way.
I'd already discovered that looking at browser output was cheap but ineffective. The same would be true of looking at the HTML wvHtml produces (see points 2 and 3 above). Using a debugger or writing a lot of C code to check the data structure directly would have been completely effective but expensive. I needed to find some cheaper way. I might trade off some effectiveness (bug-finding power) if it saved me enough time. (After all, what's the use of a perfectly effective technique if it's so expensive that you can't run half the tests you know you need?)
Understanding data
I began by skimming the documentation for Word's file format (which is astoundingly complicated) and wv's data structure definitions. I discovered that wv uses direct analogues of some Word data structures. For example, Word stores information about table rows in a structure called a TAP (TAble Properties). The wv library builds a structure called a TAP that has exactly the same fields. The same is true for the Table Cell descriptor (TC), the BoRder Code (BRC) that describes what table and cell borders look like, and so forth.
Much processing goes into producing wv's internal format from the Word document. For example, a Word file typically does not have a complete description of a table row in any single place. Instead, a "base" TAP is given, then followed by several SPRMs (Single PRoperty Modifiers) that tell how a later TAP should differ from the base. Moreover, Word's "fast save" feature causes Word to append changes to a document instead of rewriting it. This made more sense in the days when people edited from floppies. Today, it means that a Word document may contain disjoint bits and pieces of versions of a table that wv (and Word!) has to laboriously piece together.
For the purpose of checking the correctness of the wv library's data structure, I don't care about those details. All I need to do is compare the final TAPs, TCs, and BRCs to the table I originally created (to what I saw on my screen when I used Word).
(But knowing some of the details will help me plan and design tests. For example, now that I appreciate how much work goes into processing "fast save" files, I know to devote more of my testing effort to tests that look like this:
I hope never to have to learn all the details. My goal is to find all or most of the bugs while learning as little as possible.)
Understanding processing
It's usually a good idea for a tester to understand the basic flow of data through the program, and to be able to name the major components. Most often, I get that information while watching a developer draw on a whiteboard. In this case, I got it by watching the code execute in a debugger.
The most important thing I learned was that wvHtml's output is controlled by a configuration file. The file is written in XML, an increasingly popular language for representing structured data. It's quite similar to HTML. Let me explain the configuration file with a couple of examples.
In an HTML document, a table row looks like this:
<tr>
cells of the table go here
</tr>
Here's the configuration file entry that controls printing of a table row:
<row>
<begin><tr></begin>
<end></tr></end>
</row>
That looks more complicated than it is. The <begin> line says what wvHtml should print before a table row. The <end> line says what it should print after it prints the last cell of the row. Both XML and HTML use angle brackets to surround directives (tags). For that reason, the '<' for the output is written as '<' and the '>' is written as '>'. So this entry says to print <tr> before a row and </tr> after it.
How does this fit into the overall process? wvHtml works roughly like this:
But there's more to it than that, as I'll illustrate with the <table> tag, which controls how a table is printed. For clarity, I've converted the funny '<' notation into the more readable output characters.
<table>
<begin> <table width="<tablerelwidth/>%"
border="1" cols="<no_cols/>"
rows="<no_rows/>">
</begin>
<end> </table> </end>
</table>
Notice the embedded XML tags, such as <tablerelwidth/>. They are used to print variable data, instead of constant text. That is, when processing <table> <begin>, wvHtml starts by printing this:
<table width="
It then encounters the <tablerelwidth/> tag. This causes wvHtml to execute code that produces the table's width relative to the page. Suppose it's 75%. That value is substituted into the output, which now looks like this:
<table width="75.00
The <no_cols/> and <no_rows/> commands similarly calculate the number of columns and rows in the table. The final result might look like this:
<table width="75.00%" border="1" cols="2" rows="2">
Making use of the configuration file
I don't want to print HTML. For my purposes, I don't care about the relative width of the table. That's an HTML idea. It's not a value stored in a Word DOC file. <tablerelwidth/> calculates the relative width from what is in the DOC file: the offset of the left border of the table from the left margin, the offset of the right border from the right margin, and the width of the page.
What I do care about is whether wv read those table offsets correctly. So I would prefer to see my table printout look like this:
TABLE STARTS:
leftmost border offset: 23
rightmost border offset: 298
(I'm oversimplifying the way border offsets are stored, but it's close enough for this example.)
That could easily be done by modifying the configuration file:
<table>
<begin>
TABLE STARTS:
leftmost border: <tableleftoffset/>
rightmost border: <tablerightoffset/>
</begin>
<end>
TABLE ENDS
</end>
The nice thing about this is that I don't have to understand or modify the code that grovels through the complicated internal data structure and knows when to print out table information. I merely have to edit the appropriate tags. It is not unusual to look into big programs and find code that can be co-opted into serving our testing purposes.
Of course, <tableleftoffset/> and <tablerightoffset/> do not yet exist. I have to write them or persuade the programmer to write them. Although I haven't done that yet (as explained below, I'm working with other similar tags first), it's easy. To add the new <tableleftoffset/> tag, I would simply look for all the places where <tablerelwidth/> appears in the code. I would have to make changes in five places. Making <tableleftoffset/> return the correct value would likewise be simple: the implementation of <tablerelwidth/> shows how to retrieve the left offset from the big data structure.
It would be a simple matter to do this for all the elements of the table-related data structures. Importantly, I can get away with only a little understanding of the internals of wvHtml. The configuration file structure precisely guides me in what I need to do.
Programs writing programs
People who look at long lists of undifferentitated output miss oddities easily. I in particular am not by nature incredibly meticulous, so I always take care to construct output that compensates for my human frailties. I could print table output like this:
TABLE
ROW
1
6
6
0
1
16 101000000
...
CELL
1
1
0
...
But that would be incredibly error-prone. Better would be something like this:
#################### TABLE ######################
#### ROW 1
jc (justification code)=centered
dxaGapHalf=6
dyaRowHeight=6
fCantSplit=false
fTableHeader=true
tlc (table look)=Grid1 (use borders, font)
...
--cell 1,0
fFirstMerged=true
fMerged=true
fVertical=false
...
I like this better because it goes to some effort to keep me oriented (by printing row and cell numbers), it uses the official data structure names from the Word file format document (but also defines the more obscure ones), and it uses names like "centered" instead of numeric codes.
Still, I'm worried. When I'm testing something simple, like cell borders, it will be easy to be swamped with all this output. It could reduce my efficiency, it could make files of expected output less maintainable if any part of the file format changes, and I could miss problems because there's so much to look at.
A standard solution is to filter output. I might take a huge output data file and filter out everything but the cell border information. As an added wrinkle, I might check that everything that was not filtered out remained the same from test to test. (If my tests are only changing cell borders, the number of rows and columns should remain the same in each run.)
It's often useful to do such filtering with a scripting language like Perl, Python, or TCL. Of these, Python is my favorite (though I'm still a novice).
But consider: I have a program (wvHtml) that creates an internal C data structure. I write C code that prints out a textual representation of the data. I'll have to write a Python program to filter that text - which might mean reading text in and creating an internal Python data structure for it. Therefore, I should make the textual representation easy for both a human to read and for Python to read.
Well, one thing that's easy for Python to read is Python scripts. Therefore, I've chosen to print the data out in the form of a Python script that can be executed to create a Python structure equivalent to the original C structure. Below, I show what that looks like as of today's writing.
First, some notes:
Here's the output as of January 3, 2000:
# TABLE TEST OUTPUT. Config file is table.xml (This line is a Python comment.)
import ImportedTables (Load table support code.)
it=ImportedTables.ImportedTables() (The variable "it" will hold all the tables in the document.)
it.tableStart(cols=3, rows=3) ################################ TABLE 1 ###
it.rowStart() #++++++++++++++++++++++++++++++++++++++++++++++++++ ROW 1 +++
it.addRowBRC('tableTop', widthName='3', type=23, icoName='auto')
it.addRowBRC('tableLeft', widthName='1/2', type=1, icoName='auto')
it.addRowBRC('tableBottom', widthName='3/4', type=1, icoName='auto')
it.addRowBRC('tableRight', widthName='1', type=1, icoName='auto')
it.addRowBRC('internalHorizontal', widthName='1 1/2', type=1, icoName='auto')
it.addRowBRC('internalVertical', widthName='2 1/4', type=1, icoName='auto')
it.cellStart(rowSpan=1, colSpan=2) #-------------------------- cell (1,1) ---
it.addCellBRC('top', widthName='0', type=0, icoName='auto')
it.addCellBRC('left', widthName='0', type=0, icoName='auto')
it.addCellBRC('bottom', widthName='0', type=0, icoName='auto')
it.addCellBRC('right', widthName='0', type=0, icoName='auto')
...
Given appropriate Python code in the library ImportedTables, running this script would build a data structure that could be manipulated with all the power (and ease) of a good scripting language like Python. As of this point, I haven't needed any of that power, so I haven't done that. This is essentially a text dump with room to grow.
Initially, this may look less readable than the previous format, but it really contains almost exactly the same sort of information, just formatted a bit differently. I think that it's essentially just as readable - but I have been reading code since around 1976. (I should probably align the fields in pretty columns.)
Maybe you don't like this format. Maybe you prefer the example given above. But it is easy to produce that text from the Python-format data - easier than going in the reverse direction - so I could write you a simple script to show you what you wanted.
In any case where you reasonably anticipate having to do complicated processing of data, consider printing it in this programmatic style.
Discussion
Consider a movie evaluation service. Through proprietary signal processing technology, it "watches" a movie and builds up a big data structure that describes the important elements: number of car chases, number of words on screen that the audience might have to read (subtitles are a sign of bad movies), size of fireballs relative to actors visible on screen, number of bodies flying through air during explosion scenes, and so forth. It then compares that data structure to a set of rules describing what makes a good movie and issues its judgement:
or 
Thumbs
copyright www.arttoday.com.
Obviously, a lot of information is lost in this process. Suppose one of the rules about goodness is that a movie is good if it has at least two explosions and no subtitles, except that an explosion that flings five people in the air is good even if it has subtitles, and a really novel special effect involving a gun whose barrel is at least as thick as the hero's bicep counts as one explosion, and... Feeding the program the right set of movies to determine if it counts explosions correctly will be tedious and error-prone. Moreover, the tests will likely break as the rule set changes. It would be much better to "look beneath the hood" and check directly how many explosions were counted, as has been described here.
That example is extreme, but all programs suffer information loss to some degree. There's always more inside them than is visible from the interface. Part of the tester's job is to decide which of the internals should be made visible and which can reasonably be tested indirectly. That is a matter of both effectiveness - which approach will lead to the most bugs being found? - and cost.
Other sources
Len DiMaggio has an article titled "Looking Under the Hood" in Volume 2, Issue 1 (January 2000) of Software Testing and Quality Engineering Magazine
In my book, The Craft of Software Testing, I have a similar example. I built sample test support code for GNU Make by reusing debugging code.
Downloads
I will place the modified wv source on this site (as well as get it integrated into the main source base), but not yet. I have other things to do with wv first.
Be the first person to add a comment in the
Wiki Forum at page TestSupportCodeCaseStudy.
(The Forum is explained in its FrontPage.)
In this spot, the author of this page will occasionally summarize the discussion in the Forum.