One of the most common failings of XP teams is insufficient testing. XP asks for more testing than many teams are used to. But what about projects that need reliability at a substantially higher level? Are they out of luck?

XP requires that all the code be supported by extensive unit tests, which must run at 100% all the time. In addition, every requirement story must be supported by one or more acceptance tests, sufficient to give the customers confidence that the system works as needed. All this testing gives teams great – and justified – confidence in the quality of their systems.

A very common failing in XP projects is insufficient testing, especially insufficient acceptance testing. We’ll provide a rant against this mistake in another article. Failings aside, XP projects typically report higher reliability than the same teams had attained before doing XP. But what about projects that need very high reliability? What can XP offer for them?

Out of the box, XP provides very good reliability. Here are some reasons why:

  1. Unit Tests, ideally written before the code that is tested, cover "everything that could possibly break".
  2. Acceptance Tests, independently defined by the customer, test all the requirements.
  3. Whenever defects slip through the unit tests, to be detected by the acceptance tests, we recommend that the programmers upgrade the unit tests, not only to show the existing defect, but to upgrade the testing practices in general based on what was learned about the "missing" tests.
  4. Whenever defects slip through the acceptance tests and are caught by users, the same practice is used to upgrade both acceptance tests and unit tests, and the testing practices.
  5. All production code is programmed by two programmers working together. This provides one hundred percent inspection by at least one other person.
  6. In XP, code is owned by the team, not by individuals. This means that over the course of the project, essentially all the code is viewed and edited by even more programmers than the original pair who wrote it. This provides even higher levels of inspection.
  7. XP teams release software to users very frequently, ideally every couple of weeks. This ensures that the software gets plenty of assessment in the real working environment. This enables the team to build an excellent sense of system quality.

Is it possible to do even more than this for reliability? Of course. Is it necessary? Not often: XP projects in the commercial realm are reporting higher reliability than anything their teams have done before. So we’re quite sure that as a starting point, XP is quite safe.

Up to here, we’re on pretty safe ground. XP offers quite good reliability, as described above. But there are some projects – even projects of a scale where XP might fit in – that need extremely high reliability. Examples include stock trading systems that may commit millions of dollars without human intervention, and embedded medical device software that may determine whether someone lives or dies. These projects need to go all out for reliability. Can they be done with XP?

High Reliability Software

Caveat: While my teams and I have written some quite reliable software, I've never done super-high-reliability software. What I'm saying here should be even more as opinion than my usual offerings. Up to here, I've been talking about what I know about XP and what I've observed on real projects. Now we're into speculation.

Two common approaches to high reliability are inspection, and program proof. Inspection means just what it sounds like: team members inspect the code. There are several well-known rituals involving inspection, including the practices defined by Tom Gilb and by Michael Fagan, and by the IBM-developed “Cleanroom” process.

Program proof has been around since at least 1949, when Alan Turing proved the correctness of a program. I’m prepared to be corrected on this, but my understanding is that it is practical only for relatively small programs. In any real system it is probable that only critical parts would be subject to proof.

Finally, it is important to remember that there is nothing magical about mathematical proof. A proof is merely a form of human reasoning, and a sufficiently complex proof is just as subject to errors as is a computer program. And with a proof, there is no way to write tests.

It’s my understanding that for high-reliability software, the practices resorted to are very comprehensive testing, and very intensive inspection, with occasional use of proof. All of these practices are consistent with XP and could be added to an XP project without much difficulty.

Regulated Software

Medical device software, and some other forms of software are regulated by government agencies. Again, I’ve not had the privilege of working on such a project, so again, this is just educated speculation.

Approaches to reliability in these situations are no different: testing, inspection, and proof. There is often a need to trace requirements. The basic idea is to follow each requirement through the system, through specification and development, relating tests to requirements, following changes and refinements, and so on.

Requirements tracing is largely a matter of documentation, with each requirement identified and followed through the implementation and testing process. Since XP’s base documentation processes are largely informal, requirements tracing would almost certainly require a substantial increase in the paperwork on the project. It would be burdensome, but presumably worth it to gain the certainty that’s needed.