The Backlog

It all started innocently enough, with this series of tweets:

My initial reaction to Ken’s tweet was to the notion of writing user stories, which is a phrase that often results in people creating what amounts to a product specification as a series of stories. Ken’s glossary entry describes a perfectly reasonable activity where a team creates “story placeholders” for some product. As I said in that thread, translated from tweet language:

I’m not against all forms of writing down scope-related items, but I am averse to the notion of “writing user stories”, and to the notion of the “backlog”.

I’ve written elsewhere about user stories, which should be “told”, not written. for example here and perhaps most famously here. Suffice to say here that the essential communication around user stories should be in terms of talking about what needs to be done, and agreeing on how we’ll know we’ve done it.

What is the Backlog

Today, we’ll talk about the “Backlog”, which comes in two forms, the “Product Backlog”, and the “Sprint Backlog”. Our concern here is the Product Backlog. The [Scrum Guide] says lots of important things about the Product Backlog. Among those things are these:

The Product Backlog is the master list of all functionality desired in the product. Product Backlog items have the attributes of a description, priority and estimate. … The Product Backlog* includes everything necessary to develop and launch a successful product. It is a list of all features, functions, technologies, enhancements, and bug fixes that constitute the changes that will be made to the product for future releases.

Note these words: “master list of all functionality”, “description, priority, and estimate”, “everything necessary”, “all features, functions, technologies, enhancements, and bug fixes”. It seems to many people that the following is a reasonable interpretation of what’s above:

Definition: In Scrum, the Product Backlog is a master list of every single change that will be made in creating the product, each item including a description, priority, and estimate.

Now if you read the whole Scrum Guide, and you take a good Scrum course, you’ll find out that the above “definition” is wrong. The intention – or at least what’s done on the best products – is that the Product Backlog is a very lightweight, dynamic, almost casual list of the things that need to be done.

Items start out big and fuzzy, and refined items get pulled off and done, via the Sprint Backlog. From these items, the team creates ready-to-ship increments of software every couple of weeks. Each of these increments is perfectly fit to give to users, even if some important features are still missing.

Does that sound like the Scrum you’ve seen on the ground? Does it seem that that’s what most Scrum teams are really doing? I’ll tell you this: it’s not the Scrum written about here. And you’ll not have to look far to find organizations who are using a Scrum-like process to grind through an infinite list of “requirements”, with little or no regard for creating regular shippable increments of software.

Objections to the Backlog

I’m not the first person to make observations like those here. Mary Poppendieck, of Lean Software Development, has long objected to the Scrum notion of Backlog. If I understand her objections, they are to the effect that we cannot know at the beginning what we will need to build, and that we’ll need to modify what we build as we go. Now, Scrum experts will object that that’s just what is in mind with the Product Backlog: a dynamic list, moving from vague to clear as things become important and get built, and containing only the items that are important.

Scrum does describe the Backlog in that smooth dynamic way, if you read carefully. Unfortunately, many organizations get it wrong and treat the Product Backlog as a complete and comprehensive list of estimated prioritized items that must be done before the Product is complete.

What’s wrong with that, you’d like to know?

Starting out behind

The very notion of “backlog” is that it’s a bunch of work that you’re already behind on. It’s outstanding orders that must be filled. It’s work that must be done.

Some readers are saying right now, “Of course! It’s the specification for the product! It’s the requirements!” My point exactly.

As I described in The Nature of Software Development, building a product needs to be done highest value first. In an “Agile” method like Scrum, the Product Champion’s¹ main job is to choose what to do next, and what to put off, to maximize the return on the development investment.

We’re not talking about “maximize by the end”. You’re supposed to be producing a ready-to-ship increment of software every couple of weeks, and every one of these increments needs to make the best use of the investment so far. That means that in every increment, the Product Champion has selected the highest value things to do, and has trimmed them down to be as targeted as possible, to get the most bang per buck.

If you’re working from a backlog, it’s too easy to think that you have to do it all, too easy to just get into the grind of doing the next thing and the next and the next. The very notion of backlog pushes against maximizing the value of your work.

It’s all understood already

When a team has a backlog that meets our imagined definition above, most of the creativity has been squeezed out. It’s a “master list”. It covers “every single change”, all the way down to knowing how long it’ll take.

With a backlog like that, there’s no room for the Product Champion and team to look at the problem and devise creative, efficient, effective solutions. It’s all cut and dried. Just get cracking and do these features on this backlog here. Apply your “creativity” to going faster.

Product becomes project

If we have a fully defined backlog, we no longer have a true Agile Product Development situation. Instead, we have a project: a list of stuff all of which has to be done. There is no real room for the Product Champion to carry out their primary function, which is to maximize the return on investment for the development effort. The Champion does that by deciding the Product Scope, based on what actually happens during the effort.

With a fully-defined Product Backlog, the Product Champion is severely limited in doing their job.

Estimation becomes paramount

I’ve written extensively on the dangers of estimation. With a Backlog that purportedly includes all things, all estimated, it’s almost inevitable that management will focus on squeezing the estimates down, and forcing “productivity” up, so that the numbers will add up, so that all the features can be done by the deadline, which even though it was said to be negotiated, probably really wasn’t.²

The Backlog, by causing too much focus on estimation, results too often in an effort that focuses on cramming bulk features into the software, on pushing people too hard, and on making inferior decisions on what to do and what to defer.

Value becomes, well, devalued

Our false but compelling definition of Backlog refers to priority, not to value. Even more unfortunately, so does the real description of Backlog. But as I’ve written so eloquently³ in Nature, the Product Champion needs to focus on delivering the most value, not some vague “highest priority” items.

What should happen is that the Product Champion selects the most important things to do, sliced down to very thin slices, to get the best possible value by the desired date. Too often, the backlog becomes the target, everything is of highest priority, and the result is an inferior product delivered late.

Summary of concerns

Very often, this naive, mistaken notion of the Product Backlog results in projects instead of products, in excessive focus on estimation and productivity, and loses attention to what really makes Scrum efforts successful, management of scope for best return on investment.

These are not good things, and while a careful look at Scrum militates against them, they happen all too often.

But we need the backlog …

There are many arguments for having the list of all the ideas we have, all the areas the product has to cover, maybe even all the features we might want, and so on. Certainly the Product Champion has to have an overview of the product and has to have lots of details in mind. The whole team needs this overview as well.

Even the stakeholders, management and others, need to know what’s going to be done. They, however, should understand the big picture, the major capabilities of the product. It is not helpful to give them detailed backlog information, since that will never come true in any case, and they’ll expect explanations every time the Product Champion makes a decision. Since the Product Champion has responsibility and authority to make the detailed decisions, we should work to keep obstacles out of the way. Report what is built in detail: report what is planned at a level of abstraction that reflects the uncertainty.

We do need some place to keep track of our ideas. Making that place become an Official Artifact of the product development, complete and estimated, is too risky. Let’s keep our stories on PostIt™ notes or index cards in a lunch box. Use some vision statements or drawings, plus the Product Increment, to communicate about the product.

Eliminate the Backlog notion?

Would Scrum be better with the idea of Backlog removed or substantially revised? I do think so. But that’s not for me to say: that’s up to Jeff and Ken.

Would your project be better with the idea of Backlog removed or substantially revised? I do think so, but that’s up to you.

What would I do? On a single-team project, and probably on a project with a couple / three teams, I’d try something very different from the traditional Scrum Backlog. I’d start something like this:

Index Cards

I’d keep product needs, and ideas for solution, on a pack of index cards. I’d carry the cards around in a little card case. Whenever we met to plan an iteration, or to prepare to plan it, I’d pull out the cards. We’d sort them, tear some up, write some new ones, arrange them in order of our perception of the value they’d bring.

We’d pick one or two needs to work on in the next iteration, and during planning I’d invite the team to create lightweight innovative ways of serving those needs. We’d build those things as our next product increment.

Product Vision

I’d have a few drawings of the problem space and the solution space as the team presently understood it. I’d have a few paragraphs describing what we had in mind. I’d have an elevator pitch for the product, covering needs and current status. I’d have another similar pitch for every major problem/solution chunk in the product. If need be, I’d publish those as documents or slides every now and then.

Product Increment

I’d produce an integrated, shippable version of the software, every week or two. This would be my primary indicator of progress. If people had a clear view of the product and how it was improving, they’d be far less concerned with the infinite list of what we might do someday, and a lot more interested in the very finite list of what we’ll do next week.

Regular Live Product Review

I’d have a live product review at the end of every iteration. All our key stakeholders would be invited to this meeting and expected to attend. We’d demonstrate the live product and attendees would discuss progress and what we should do next.

Every way I could, I’d try to make the Product Review visibly the main way – almost the only way – to find out what the product status is and to influence it. Maybe I’d go so far as to have my status report consist of little more than a video of the Product Review, making it clear that this is where you find out what we’re doing, and this is where you give us your guidance on what to do. I’d make it clear that the people at the Review are listened to, and try to make it clear that if you’re not at that meeting, you’re falling behind.⁴

Some stakeholders are “too busy” to attend these meetings. Fine with me. They can send an empowered delegate, and I do mean empowered, or they can reap the results of not being involved. I’d do everything I could to make the Product Review the only way to guide this product.

What about your backlog?

No doubt some people would ask for a list of all the things we planned to do. Our answer would come in the form of the elevator pitch for the areas covered. We’d tell them what we knew about the solutions. If they were unsatisfied, we’d offer to work more on that area, if the Product Champion asked us to. We’d make clear that that effort would push some other effort back, and encourage the stakeholders to sort out for themselves what stories would be most valuable right now.

Sometimes, they’d ask us to keep their item on a list. We’d reply in the light of “Petition the King”. If your idea is of value to you, the Product Review is the time to bring it up and make your case for it. You’re the one with the best case: come to the meetings, see how we’re doing, push for your idea.

Could you possibly get away with this?

Well, yes, I probably could, and likely so could you. I’ve had successful projects with no software on view until the end, no concerted effort to keep people up to date on real progress, and a list of features some of which never got done. Quite likely, so have you.

So I’m rather sure that starting this way, and adapting as the situation called for, could keep me a very long way from a conventional backlog full of all my “requirements” and their estimates.

And I would try, really hard, to make that work.

What should you do? That’s up to you.