The Purpose of Beta Testing
Most commonly, we do beta testing because someone somewhere wants a final check that everything is OK. In XP terms, we fear that if we don’t beta test, something bad will happen. Beta testing is done to alleviate that fear.
One of the XP values is Courage. If the organization feels enough fear to want to do beta testing, then I’d like to suggest that there is not enough courage. The response isn’t just to hold our chin up and say “Be brave, little Piglet,” or “Damn the beta testing, full speed ahead.” But the response might be to examine the information that comes back to those who hold this fear, and try to get it sooner.
Information from Beta
Beta gives us back two basic dimensions of feedback, which I’ll call Defect Feedback and Feature Feedback.
We might find actual defects in beta. The system might not run correctly on some user configurations, or there might just be defects in the software. The technical term for this is “bad”.
Beta testing, in its common form, is done very late in the development cycle. We think we are done and just want to be sure. This is a very bad time to be finding defects. Whenever a defect escapes from a coding cycle or from an iteration, an XP team improves its tests and its practices to prevent that kind of escape from happening again. This is an application of the XP value of Feedback.
A defect found during beta testing was probably injected, on the average, halfway from the last beta test until the current one. By XP standards, where we expect testing feedback every couple of hours, this is seriously open-loop, not the best case of the XP value Feedback.
Therefore, a wise XP team will do pre-beta testing or on-site configuration testing, or take whatever measures are necessary to reduce the chance for Defect Feedback during beta. We want beta to be a non-event.
Even if the software works as planned, we might also do beta testing to be sure that it is what the customers want. We might not have built quite the features they asked for, it might be harder to use than they like, and so on. This is, in essence, feedback to the XP Customer about how good a job she has done.
As with Defect Feedback, a problem discovered in Feature Feedback was probably injected, on the average, a long time back. As with programming defects, this feedback is too slow to be really useful. The wise customer will find ways to be sure, long before beta-time, how well her decisions match up with the needs and preferences of the buyers and users. The XP Customer, too, wants beta to be a non-event.
Preparing for Beta
In general, we should think about all the bad news that could come back from beta testing, and get that information much sooner, by any means possible. It’s difficult to be more specific, since what we do depends on what the risks are on our specific project. Here are some starting examples just to get us thinking:
The program might not run on all configurations. Should we run the program on lots of configurations? Get some of our users to try early versions of the system? Select components which are known to work well on more systems? All of these and more?
Some customers might not like some features. Should we check with lots of customers to see what they like? Get them to try to features to get more information? Provide some way to configure feature preferences that are in conflict or deal similarly with important minority preferences? All of these and more?
The specifics are always different, and I’m confident that you are up to it. You are the experts in your own situation. The general rule, I suggest, is always the same: Make the beta-test period into a non-event. Turn beta testing into business as usual.
Team Activities During Beta
We have done our best to make the beta test a non-event, but for whatever reasons, the beta test is still being done. What should we do during this period? Commonly in this period there is a “code freeze”. Let’s dispense with that.
Team Wants a Code Freeze
A team-driven code freeze is often an act of fear. We freeze the code because we are afraid that if we change it, we will break it. We have probably all been in that situation, and it’s not a pretty sight. However, fear is not the XP value: Courage is. An effective XP team has done so much testing, and so much beta preparation, as not to have a fear of changing things. They do not need to let themselves be ruled by the fears that call for code freeze.
Instead, the XP team might operate in normal mode. Normal mode for an XP team is another iteration. The XP Customer picks the next few important things to add to the product, and the team adds them. No big deal. Two weeks later, they ship running tested software with all the beta test features in there, and a few new features as well.
I believe that this, “business as usual”, is the right default for an XP team during a beta test phase. If the team is not comfortable doing business as usual during beta, then there is something to learn from that discomfort (see fear). Let’s learn it, ideally long before beta time, so that by beta time, we don’t have that fear any more.
Business Wants a Code Freeze
Now the business people might have the same fear and want a code freeze even if the team knows it does not need one. One of the XP values is Communication. If management has unbased fear about what we are doing, then we have not been communicating with them well enough. If management’s fear is well-founded, and we don’t share it, then again there is a communication issue. Long before beta time, let’s be working that communication link to be sure that we are on the same page.
Business Needs a Code Freeze
There might also be a legitimate business reason for a code freeze, rather than a fear-based reason. For example, the next features might be being held back for the next release for marketing purposes. Or the product might be highly regulated or under a strict contract and we might be constrained from putting in new capability. I believe this to be extremely rare in practice. More commonly, everyone wants new features if they are going to work. But it could happen. What should we do?
Well, for whatever reason, they insist on a code freeze. Therefore we will have a code freeze. What should the programmers do during the beta period? Here are a few choices:
Go on vacation. The product is essentially done, we are confident that it all works and is as required. Perfect time for a break. Let the team rest. Work on something else. Maybe there's another project going on that isn't in beta and this would be a good time to put some effort into it. The Customer can readily decide that. Branch or equivalent. There might be important features to put in for the next version. If the code is frozen, and we want to program on the next version, in some sense we need to branch the code. Remember to merge back as soon as possible.
In summary, avoid code freezes as a response to fear. If there are business reasons to freeze, try to avoid working on this product until the freeze is over. If you need a branch or equivalent, bite the bullet and do it.
Handling Feedback from Beta
If we do beta testing, then with the best of will and effort, there is still a chance that we’ll get some feedback to which we wish to respond. How should we do that?
Business as usual. The feedback has either identified a defect, or identified a feature that needs changing or implementing.
Schedule the changes as always: estimate them, and let the XP Customer schedule them in the next iteration. In the case of a defect, proceed as always: write a test to show the defect, fix the defect and prove it by running the test, then reflect on what has been learned, so as to improve the process. In the case of a new or changed feature, implement or change the necessary customer acceptance tests, build the feature in test-driven fashion, run the tests, and release.
Business as Usual
The flavor of an XP project should always be that whatever happens, it is business as usual. We get to that state of affairs by learning from experience, and by looking forward a bit, so as to taste our fear of the future and then get the necessary feedback information in good time.