Sure, test automation is a good thing. But we can't, and shouldn't, automate them all. Why then, ask people to "automate all tests"?

Slogan? Logical Impossibility? Foolishness?

A recent discussion on the agile-testing yahoo list has become a bit heated and perhaps less productive than it might have been. All my fault, I imagine. The topic is the phrase that some of us Agile practitioners use saying that “all tests should be automated”.

Now it’s clear on the face of it that no one really automates all tests, and that no one really could automate all the tests that are needed, and that it would cost a huge amount of time and money if they tried. Furthermore, there are obviously many important examinations of the product that can only be carried out “by hand”, under the watchful eye of someone with good intuition, good product knowledge, and a generally skeptical nature.

In the face of these truths, how could anyone ever say “all tests should be automated”?

All these notions and more have been offered, and in that charming way I have, I’ve pretty much rejected them all. Why? Is it that I’m a fool, or a jerk? Well, I may be those things, but that’s not why.

Stefan Steurs suggested that many of the XP sayings sound like slogans to him. I agreed that they are like slogans. I’m not sure whether he thinks slogans are bad, good, annoying, or useful, but surely neither he nor I think that slogans are precisely true.

Ideals

It occurred to me here in the middle of the night that those sayings are not exactly slogans, so much as they express ideals. Let’s talk about whether “automate all tests” is even a reasonable ideal.

On the one hand, it couldn’t possibly be a reasonable ideal, as Michael Bolton has so cleverly established by pointing out that if all tests were automated, there couldn’t be any manual tests and there are clearly some tests that can only be done manually. Therefore, he apparently concludes, the suggestion to automate all tests is meaningless, null and void, without merit, empty of value, and generally not such a good idea.

As might be anticipated, I hold a different view.

Excellence

Probably most of my interlocutors would agree with the following proposition, though I haven’t tried it on them:

It is of great value on a software project to be truly excellent at automated testing.

What does this mean? The great philosopers Bill and Ted advised us to be excellent to each other. While this is “excellent” advice, it isn’t entirely clear how to be excellent, rather than merely good. Such is the case with automated testing, as well.

Many projects – even some of my own – don’t have much in the way of automated testing. A quick examination of what is happening on those projects will usually show places where there would be great value to more automated tests. It’s easy to get a team to see that they “could use a few more automated tests”.

That’s a good thing, isn’t it? It’s a step on the road to excellence.

Yes, it is a step, and yes, it’s a good thing. On a given day, I’d try to take some pleasure and pride from having helped someone take that step. Other days, not so much.

How Good Do You Want to Be, and When?

I’ve been feeling a little dissatisfied lately, when folks I work with take a few steps in a good direction, and then stop stepping, deciding they’re good enough. Now, hell, I’m surely not perfect, and in many areas probably don’t measure up to anyone’s standard of good enough, yet you don’t see me, marching forward in all directions, becoming smarter, healthier, better looking, more modest, and more excellent at test automation. So who am I to talk about Good Enough not being good enough?

Well, I’m just me. The question I’ve recently been asking myself and those around me is, “How good do you want to be, and when?” I decide for myself: you decide for yourself. If, in some area, our answer is “I want to be excellent,” then here’s something we need to understand and embrace:

Excellence is a Matter of Inches

The transition from good enough to excellent occurs at the high end of the range of capability in every endeavor. The fastest runner is fractions of a second faster than the second best, and not very much faster than even the mediocre ones. The world high jump record is about 7.9 feet; mere high school students have cleared 7.15 or more.

Every realm is like that: at the high end the differences come in the last little bit.

We can choose where to be excellent, and where to be good enough. If we choose to be excellent at automated testing, I think we’ll find that, from where most of us stand today, excellent lies so close to “automate all tests” as to be indistinguishable.

So that’s my advice: if we want to be excellent at automated testing, we should set out to automate all tests. We’ll probably never get there, but we might get the Silver and Gold along the way.