We've been working for almost two full days now, and it seemed time to refine some stories. We found some conflicts between ourselves as programmers, ourselves as customers, and ourselves as advisors to projects.

Story Planning

The snow didn’t accumulate, and no other Celestial Obstacles appeared, so we met in Brighton. Our plan was to address stories, and except for a little messing about with VNC, and whining about the fact that our computer’s folder structures are different1, 2, we wrote stories. Well, almost.

Chet started writing story cards, and I started objecting to them.

Center of Pattern

Chet wrote “Center of Pattern”. I asked if that story was done (we do have code, and we’re sure it works.) He said that it was. I asked how, as customer, he knew it was done. There’s the rub.

We have perfectly good JUnit tests that tell us that we can calculate the center of the pixel pattern. But there is no user-visible output, nothing that we in our role as project advisors would bless as an automated customer test. Now frankly, if this was just Chet and Ron writing a product, I suspect we would consider the JUnit tests sufficient, because we can in fact mostly understand Java code. In our role as authors, advisors, and acting as Normal Human Customers, we can’t accept those tests as suitable. They amount to asking our customers to trust us. Our recommended approach is that such matters should be expressed in tests that the whole team can understand, and shown to run.

Identify Hits

Chet wrote: Identify Hits. It turns out that overnight, he had taken our 3-foot square target paper and counted the hits. There were 1237 if I recall. Since there are nominally 410 pellets in his loads. this tells us that he blasted this target three times, not once. There are some 5600 black pixels in the picture, which means that there are between four and five pixels per pellet hole, on the average. The “Identify Hits” story means, essentially, that we need to consolidate the contiguous black pixels into one hit, and probably also means that we need some rule of thumb to decide how many pellet holes are in one big blob of black. I asked how we were going to test that.

Doing consolidation seems a bit tricky all on its own. Deciding how many pellets made a hole is a matter of human judgment in the best case, even looking at the paper. Looking at the picture, there’s no telling at all, as you saw in the previous article. Even if we had an incredibly high-res photo, the problem would still be a difficult image-processing one, and that’s not really where our customer wants to go, especially not at the beginning.

Still, what’s the test? If I write a pixel-consolidation method and come up with the contiguous blocks of black, and there are 1199 of them, does that agree with Chet’s count or not?

Pattern Density

Chet wrote: Pattern Density. We have a spike on that: it counts Hits in each of a number of rectangles spanning the picture. Chet bought a lovely five-foot aluminum ruler and gridded the target, and counted the holes in each grid. Holes on the line he counted in the first grid he tallied that included that line. (There’s an algorithm that will be hard to replicate.) So for the test, Chet has a matrix of about 10x20 numbers, tallying the hits in each cell. I asked if we had to hit them exactly. He said that, no, since we won’t always know how many pellets made a single hole, our numbers might vary. I added that since our computer grid won’t fall on the picture the same way his lovely ruler fell on the paper, we’ll probably count a few in neighboring grid cells.

For a moment, we were thinking about a simple FitNesse test with a grid of 200 numbers and we’d just create the same grid in the computer and compare. Shouldn’t be too hard a fixture to write. But the fixture needs to be a bit forgiving. Should it be that it accepts a constant number of pellets above or below the value in the grid? Should it be a percentage? We don’t know.

Circular Density

Talking about pattern density, Chet added some story detail. The standard way that you do a density calculation by hand is that you draw a 30 inch circle centered on your shot pattern’s center, and an inner circle of diameter 21.21 inches3. Then you divide those into quadrants, and you count the pellets in each quadrant. The counts should be approximately equal. Chet wants this notion improved, since we have a computer to do the job. He wants three circles, not just two, and he wants the circles divided more times, probably into octants rather than quadrants. And he observes that you probably really want more shot toward the center than toward the outside, but that remains to be seen. In any case, you want to know what your gun is doing.

image

I gather that part of the process a shooter goes through with this is to try various loads, to decide what size shot and what size load gives him the best results with his particular gun. That’s great for us: more pictures means more revenue. We didn’t get there yet, but surely there will be stories about using the computer to compare the pictures and draw conclusions or provide information that will help the shooter make that decision. For now, we just have stories around a single picture.

We digressed to talk about how to calculate the pattern density in this circular way, including a little digression on Pi R Squared, and two times the square root of 112 and such. As you can see in the picture, the segments in which we have to count holes are pie wedges and circular arc-shaped things: bent rectangles. That’s just too interesting a problem not to digress on, so we did.

We note that a given arc shape is the area between some angle and another on the grid, and between some radius and another. Hmm, radius, angle. Hmm, polar coordinates. Sure enough, I just knew it: our customer came up with a story requiring us to convert the hits to polar coordinates. I knew that trig course would come in handy someday.

Do We Really Need These Tests?

We are programmers, and we understand our JUnit tests. And we are our customers, and we can trust us. Therefore, we argued, we don’t need to build automated customer tests for these things.

But we are doing this not just to build a product, but also to learn how to solve this kind of problem with these tools, and to share that learning with our vast audience “out there”. And they will torture us mercilessly if we don’t follow the rules that we set out for them to follow.

We talk with teams all the time who struggle with this notion. “The tests will slow us down,” they say. “Do them anyway, you’ll be glad you did, and if you don’t, you’ll regret it” we reply in that kindly way that we have. And sure enough, if they do, they’re glad, and if they don’t, they regret it.

“But our product is different,” I said. “It is reports and graphs. And because we accept that the pixels aren’t pellets, the computer results will vary a bit from the human’s eyeball. Therefore we can’t have tests that really verify what the customer wants.”

“We’d never let anyone else get away with that,” Chet said. “And besides, they’ll make fun of us if we don’t do it.”

We discussed this quite a bit. I was feeling very afraid of trying to write some kind of fixtures that would process the kind of information I know this system is going to produce. I don’t know how to do it and it will take valuable time away from programming the features. (Of course the features may not work, and I know that perfectly well, but still, I fear writing these tests and I don’t fear writing the code.)

We came to an interesting conclusion. We actually believe that some of the acceptance tests we devise may well be a waste of time, in that even a customer who couldn’t read JUnit (but who had come to trust his team) would decide that the cost of the tests was too much. But this belief is based on fear. We fear that these tests will be expensive to write, fragile and expensive to maintain, and that they won’t tell us anything.

At Agile2006, we led a Discovery Session called “Crushing Fear Under the Iron Heel of Action”. The session had one central point: when facing fear, don’t run away, think of something to do that will settle the fear, and do it.

Therefore, what we are going to do is to commit ourselves to customer tests that a Normal Human Customer can understand. We’ll track their cost (and our fear about their cost), and we’ll pay attention to how helpful they are. In addition, we’re not going to limit ourselves to FitNesse tests. FitNesse is a good tool, but it isn’t the only tool. We are testing graphical algorithms, and looking at a table of numbers isn’t all we need to do. So we’ll keep our eyes open for other ways of testing.

One of those will surely be the “Golden Master” test. We already have one of those in the JUnit tests:

   @Test
    public void useRasterToCreateShotPatternBiggerFile() {
        ShotPattern  shotPattern = new ShotPattern(folder + "\\PatternProject\\PB270011.bmp");
        assertEquals(new Hit(123,281), shotPattern.CenterOfMass());
    }

This test reads the big BMP file of the real target. It asserts that the CenterOfMass is (123,281). You can be sure that we didn’t hand calculate that number. We ran the test, looked at the answer, looked at the picture, marked where that point was, and accepted the result.

Note: We talked about this test and technique a bit. Back many years ago on a project best left unmentioned, we added a feature to the acceptance testing tool that allowed the customer to look at the results, and press a button indicating that they were correct. This turned out to be less than ideal: they were prone to accept the computer’s answers at a glance, without really checking. We needed to put them through a bit more work. Tests of the Golden Master kind will verify that things don’t change quite nicely. But they don’t really show whether things are correct.

As I write this, I just thought of another way to verify this result. There are 5000-odd pixels in the Hit list for this picture. It would certainly be possible to spit out the X’s and Y’s in some format and sum them, if not by hand (volunteers?) but at least with some other program. If we got the same result, we’d be even more sure than we are now that we have the right answer. Maybe we’ll do that. In any case, the test above is a “Golden Master” test, in that we checked the result somehow, and now accept it as the truth.

A more common form of Golden Master, and what we had in mind as something we might do, is a case where we produce a picture or a report, the customer hand checks it, and then we use that as a value to compare against the system’s output in the future. If it ever changes, we know we have to recheck to see if it’s still good. We expect that we’ll use that approach to create tests for pictures that the system creates.

Bottom Line

We didn’t get as many stories as we need, because the team needed to talk about testing. We agreed with our customers that we will dedicate ourselves to doing real Customer Acceptance Tests, understandable by Normal Human Customers, and that we’ll pay attention to how difficult they are to do. We’ll consider more kinds of tests than simple FitNesse tests. And, of course, we’ll report all this to you.

We can’t meet on Wednesday, as Chet’s wife Sue is going to be home from work, and I imagine they are probably going out Christmas shopping for me. So what we’ll do separately, time permitting, is that Chet may spike a FitNesse test into our code, and I’ll play with the polar grid. Or, perhaps we won’t do anything like that at all, and will just take the day off. Stay tuned.


  1. We have some sample files that our tests use. Since our folder layouts are different, the naive approach of putting the file paths in the tests means that one of us will edit the paths to make the tests run. Then when he checks in, they break for the other guy. There is probably some standard way in Eclipse / Java to set up a kind of path or something that could be different on our two systems, but if there is we haven't been able to search it out. If you know a good way, an email note will be helpful. Thanks!
  2. Our use of Eclipse workspaces is different. Chet creates a new workspace for every project; I read something recommending keeping all one's little projects in one workspace. In discussing what we do, we realize that neither one of us has a good sense of the proper working relationships in Eclipse between workspaces, projects, and files. If you know of a good resource to read on that subject, and other such Eclipse tricks and traps, please pass that along as well. Thanks!
  3. This is, of course, the diameter of a circle with half the area of a 30 inch circle. Proof left to the reader.