Jeff Sutherland said today that it’s not done if it’s not tested. You won’t believe what happened next!

Jeff Sutherland is the creator of Scrum. As long as I’ve known him, he has always said that XP-like practices are necessary if Scrum teams that build software are to become what he calls hyper-productive.

The exchange here is typical: I see and engage in similar ones every day. Paul Klipp determines that if this is a big problem, then Scrum has failed. People like to say that Scrum has failed, or that Scrum should die in a fire. Well, yes, and in that sense, every good idea has failed.

Democracy has failed when idiots and evil people get elected. Religion has failed when people do bad things. Medicine has failed when people don’t take their meds. Baseball has failed when people don’t enjoy hitting paper-stuffed socks with pine sticks found on the ground.

It’s fun to say that. And, in my opinion, it is a major cop-out to say it as well. There are a lot of reasons why “Scrum doesn’t work”, but the more important question is why what a team is doing doesn’t work. There are a lot of reasons, and usually Scrum is not one of them.

Scrum is based on a very simple and very strict principle: Inspect and Adapt. This idea has many names, and as Dave Snowden tells us with his Cynefin model, the idea needs to be applied differently depending on the situation. But it all comes down to looking at what’s happening, and changing what we do to try to make it better.

Let’s get down to cases.

Scrum does not say “you have to test your code inside the Sprint”. What does it say? Scrum says that at the end of every Sprint you must produce a product Increment. Regarding the increment:

The Increment is the sum of all the Product Backlog items completed during a Sprint and the value of the increments of all previous Sprints. At the end of a Sprint, the new Increment must be “Done,” which means it must be in usable condition and meet the Scrum Team’s definition of “Done”. It must be in usable condition regardless of whether the Product Owner decides to actually release it. (Scrum Guide)

Note that last sentence:

[The Increment] must be in usable condition regardless of whether the Product Owner decides to actually release it.

In addition, under Definition of Done, we find “thoroughly tested”, which pretty much settles it:

Each Increment is additive to all prior Increments and thoroughly tested, ensuring that all Increments work together.

The Product Increment Must Work.

It seems to me that Scrum is pretty clear about this: every Sprint, you must release a product increment that is tested, actually works and can be used. If we’re not doing that, we’re not following Scrum’s advice. And Scrum isn’t at all vague about this. You can’t pretend you didn’t notice that the Increment needs to be usable.

Untested Software Doesn’t Work.

Let’s face it: untested software doesn’t work. Even if, by chance, it did work, without testing we don’t know for sure whether it works. If we don’t know whether it works, we can’t certify it as usable. Therefore we need to test it. As Professor Brehm used to say: “Punkt. Schluß. Neuer Absatz.” As Dad used to say: “That’s all she wrote.”

Sprint Retrospective

Dudes! We got no Product Increment! What’s up?

Suppose, now, that our Scrum team is going merrily along not producing a working product increment. We really ought to know this: the increment is one of only three required artifacts in Scrum, and it’s the most important one. We’re not doing what we set out to do. We have to Inspect and Adapt. We have to talk in our Retrospective about it. It’s unlikely that there’s anything more important. “No working product increment” simply must make it to the top of the list of deviations.

Scrum says to inspect and adapt our process to improve it. Here’s a critical improvement that’s needed. What do we do?

Well, it’s obvious, isn’t it? We decide that for us a working product increment just isn’t practical, and since we are all about practical, we drop this central requirement from our version of Scrum. We inspect and adapt into not doing Scrum.

Sorry, team, we own this one. Scrum advice is clear: produce a working product increment. To know it works, we must test it. We don’t. Scrum made the problem visible and we decided not to do anything about it. It’s on us, not on Scrum.

Not Doing Scrum

So, on the one hand, if we’re not testing inside the Sprint, we can’t know if the Increment works, so we can’t ship a working Increment, and therefore we’re not doing Scrum. (We’re also setting ourselves up to fail, but we’ll find that out soon enough.)

Now this is where we pull out “No True Scotsman” and various other excuses that make it OK that our so-called Scrum project wasn’t shipping a working Increment every Sprint and it’s not our fault. Waaah. It’s Scrum’s fault!

What were we expecting? That the Scrum Inquisition would come swooping in in their red outfits and whap us upside the head about our increment? Even if they had shown up, we’d have made our same excuse: “That doesn’t work for us. Our situation is special.”

Well, if the Scrum Inquisition did show up in their cool red outfits, they could have at least ripped the Scrum merit badges off our shirts and set back our “It has been 221 days since we’ve been doing Scrum” sign. It has been zero days of doing Scrum. Zero!

What has “Scrum” done to make this clear?

Every project has the Scrum it deserves.
– not Joseph de Maistre

While I do think we get the project we deserve, by choosing what to do (or where to work), I also believe that the Scrum community (whatever that is) has a real responsibility to help people succeed with Scrum. That job has not been done perfectly, and never will be. I’m inclined to say that it could be done better, but there’s a lot of evidence that testing is central to Scrum:

  • Jeff Sutherland, Scrum creator, has said repeatedly and consistently that XP-style practices, including testing, are necessary for success.
  • The Scrum Guide explicitly says that the Increment must be thoroughly tested, and has done for some time.
  • The Scrum Alliance Core Scrum page says that the Increment must be fully tested.
  • The famous “ScrumButt Test” explicitly says that features must be tested and working at the end of a Sprint.
  • The Scrum Alliance offers its Certified Scrum Developer program, which teaches Dev Team member the essential skills behind delivering a product increment every Sprint.
  • Many Scrum authors make it clear in their writings that solid technical practice is needed in order to deliver the Increment.
  • Many Scrum teachers also emphasize the need for technical practices including testing.
  • Many “Agile” authors and coaches make these same points, offer these same teachings.

What more should we do?

When I started this article, I expected to say that while Jeff and others have always been strong on testing, the topic was glossed over and more work needed to be done. However, I’m finding explicit references to testing inside the Sprint everywhere I look.

Could testing be emphasized more? Sure. It would be at the expense of something else, of course, but more could be said. If Jeff is right that this is one of the biggest problems in Agile, it surely needs more emphasis. At the same time, I’m really not sure what “should” be done.

(Spare me the inevitable rants about how Certified ScrumMasters don’t know enough and that certification is a Scam, and so on. Asked and answered.) Yes, there are issues. People need to know more. Everyone in the “Agile” business is working hard to help people understand what the need to do and how to do it.

Bring it down to cases. First of all, what might you do that would be helpful? I’d say that it’s not helpful to do the ever-so-enjoyable “Hahaha, see? Scrum fails” routine. Even if it were true, it wouldn’t be helping. Helping would be saying things like “Testing is a key part of Scrum. You can’t do good Scrum without it.” Helping would be writing articles about how to do testing in an Agile situation, or how testing improved your situation. Things like that.

Help make the world better. Offer better ideas, not mere slams at good ideas. Got some ideas that could help? Get them out there. You could even write me a note with suggestions.


Scrum says the Increment must be tested. Has always said it. Yes, Scrum/Agile people could say more and they could say it more loudly. But there’s no excuse for not knowing that’s our advice, and not much excuse for not testing inside every Sprint.