We'll Try

“We’ll try.” These are the saddest words a programmer has ever spoken, and most of us have spoken them more than once. These words are often the preface to months of grueling effort against a deadline we know in our heart we cannot make. At the end, we come up tired, burnt out, beaten, and short. Management hates us, we hate ourselves, our families don’t know us any more or have fallen by the wayside. The software, if it works at all, is nothing to be proud of.

Oh, there have been exceptions. Successful products have been launched this way, and there is a certain pride in having gone through hell and survived. We have to believe it was worth it, if the alternative is to believe we wasted a big chunk of our lives.

There has to be a better way. Here’s one that couldn’t really happen.

Suppose you knew everything they were asking for, and suppose you knew how long it would take your team to do every one of the things they were asking for.

Suppose you knew that, and you weren’t afraid of the truth. Suppose you wrote down everything they were asking for, maybe on little cards, and you went in to them and laid your cards on the table.

Suppose you said: Here’s everything you have asked for, and on each card I’ve put down how long it is going to take to get done. I’ve broken them down into three-week periods, and in each period I’ve put as many cards as will get done in that period. As you can see, it will take 14 periods to do all this.

You lay a pencil vertically between two of the columns, and say: Here’s the date we want to deliver. We have too much to do. Our job now is to put the cards we want the most on the left side of the pencil. When we put a card over there, we have to remove a card with the same number on it.

They rant. They rave. They call you names. Secure in your perfect knowledge, you say, This is how long each of these things will take. To get the best product by your date, we need to put the cards we want most on the left side of the pencil, removing cards with the same number.

They threaten your job. You say, This is how long it will take our current team to do it. Maybe if you fire us, you can find, recruit, hire, and train a team that will get it done sooner.

Bask for a moment in how calm, how strong, how totally cool and heroic you would be, because you know how long it will take.

But that couldn’t happen, could it? Yes, it could. We may not be able to do as well as the lucky devil above, but it turns out we can do pretty well. Here’s how an XP team does it:

For a moment, get in touch with that feeling you have when you’re just coding along. The world goes away, you code and test and test and code, and quickly you’re done with whatever it is.

A day of that is what we call a PerfectEngineeringDay. For a lot of your tasks, you probably have a solid feeling: If you guys would just leave me alone, I could do that in two days!

Cool. With that in hand, we only need three things:

1. We need to know all the tasks there are;
2. We need to have that solid feeling for each of them;
3. We need to know how many real days it takes to get a PerfectEngineeringDay.

Armed with that information, a stack of cards, and a pencil, we can estimate how long any project will take!

It couldn't be that easy!

It isn’t exactly easy, but it isn’t hard. Briefly, here’s how an XP team goes about it.

First, we have to have the UserStories on cards. It would be nice to have all of them. It’s important to have enough of them, and to write placeholder cards for stories that don’t exist yet.

The entire team, customers and developers, goes through all the stories. Customers explain what the system has to do, and the team brainstorms quickly how it might be done. Estimate each card in programmer weeks: 1, 2, or 3 weeks of one programmer’s time.

If a story seems much less than a week, batch it with a few other small ones. If it seems more than 3 weeks, take that as a sign that you don’t understand it. Get the customers to break it down into two or more stories and explain it again. Repeat until you have all the stories estimated.

Now no one believes we can do all those stories in the time we estimate, because there is so much else to do that we haven’t counted. We’ll start by assuming we are off by a factor of three. Yes, three! We call it LoadFactor.

Pick an iteration size (C3 uses 3 weeks) and figure out how many programmer weeks there are in an iteration: 3 weeks X N programmers / 3 LoadFactor.

Voila, you plan for 1 week’s work done in every iteration, for each of your N programmers!

Start arranging cards into groups of N weeks’ work. Each of these piles will take (we estimate) 3 weeks to get done. Count the piles, check the calendar. That’s your prediction for when you’ll get done.

Of course no one believes this. But we’re going to do it again and again, as we go along in the project. And we’re going to refine our ability to estimate, and we’re going to learn more about how the system works, so we will know more about how long things will take. A little way into the project, we’ll be really good at this. Better yet, because we will be tracking this performance, management will come to know that we are really good at it, and they’ll start believing our estimates.

At this point you might be asking yourself some questions …

What if you dont have all the stories?

You can be sure you don’t have all the stories. It’s important to at least have placeholders for all the big ones. Spend a little time brainstorming, customers and developers together, about what else might be needed. Make cards for the things that make sense. Estimate them like all the others.

How do you get estimates?

Estimating is scary. Assume for now that no one will know but you what you come up with. We’ll talk about how to deal with errors shortly.

With all the developers together, estimates will tend to average out. And remember: we have the Commitment Load Factor to give us some slack, and we will be revising the schedule many times as we go along.

Developers divide up into teams of two or three. Each team looks at each story. Reflect on how you are going to do it in the system, using your experience with the System Spike Solution to guide you. Talk through an option, estimate how long each step would take.

Consider alternatives. If you think of an easier alternative, or someone says they think they can do it in less time, take the smaller number. When the other teams look at a card that has already been estimated, they can reduce the time but not increase it.

If you can’t estimate a story because there’s something you don’t know, something not yet figured out, put down the one you can’t estimate and pick up the precursor and estimate it. Get back to the dependent story.

Repeat until all teams have looked at all stories. If you take 10 minutes per story, you can do 150 stories in 3 days. You’ll probably wind up going faster than that after the first few.

How do you explain load factor?

We recommend a commitment load factor of 3. That is, we assume that every week of time you estimate for development will take three weeks of real time.

This number is a rule of thumb. An experienced XP team might have a better number.

Here are some of the things that make up the number. There are probably more.

1. You probably don't have all the stories, and some of the ones you do have will change as customers learn more about what they really need;
2. There will be meetings, reports, writing, support, testing, planning, various other activities that mean no one will really be able to program 8 hours a day day in and day out.
3. You don't know your real Load Factor yet, and this is a fairly decent starting point.

We can't tell management our real estimates!

Some teams are afraid to tell management their “perfect engineering” estimates for the stories, for fear they will be held to those estimates instead of the loaded ones.

If this is your situation, use “perfect engineering” in your head, but call the estimates eXtremeProgrammingUnits (XPUs) or something. One project called the estimates Gummi Bears, but it is probably better for the developers to think in terms of their own perfect time.

Then it is simple enough to say to management: “In each three-week period, each developer can do one eXtremeProgrammingUnit. eXtremeProgrammingUnits are carefully calibrated estimates of difficulty. We’ll be measuring our rate of delivering XPUs as we go along, so you’ll be able to track how we’re doing.”

This can't possibly work!

You’re asking how this could possibly work. The amazing thing is that it actually works pretty well, even for your first estimate of the project. But what makes it really work is that you do it again and again.

When you present your first Commitment Schedule, explain to management how you got the schedule. Then tell them that you do not believe this schedule, and that neither should they. You go on:

"Many things can, and will change in the course of this development. Customers will change requirements, some things will turn out to be easier than we thought, and some will be harder. That has happened in every project we have ever done, and it will happen this time. "The difference with this project is that we will do this schedule every nine weeks as we go along, and we will report the results to you. We will refine all our estimates of the remaining stories, based on what we have learned in the preceding iterations. Each time we get together, we will all see how many stories are done, and how many there are to go. "Each time we get together, expect to see that we are closer to completion. Expect also that the date may move in, and it may move out. But you will be able to see exactly what our estimate is, and that will enable you to make good decisions about the project. "We're confident that we can give you quality information about how we're doing, and we're confident that with that information you will have the best chance of helping us be successful."

And you will do just that: you will observe your own performance, and you will estimate the stories over again based on what you know, and every two months your ability to estimate the schedule will get better and better.

Even better, customers and management will learn that you are telling the truth as you know it; they will learn that your estimate of the schedule is the best they can get; they will learn that they can help with the delivery by providing you what you need, and by reducing scope judiciously to help you make the date.

Not with my management!

Sometimes teams believe that even with the best knowledge of what the schedule will really be, their management is so draconian, so unenlightened, so evil, that the process can’t possibly work.

Well, if it’s really that bad, I’d advise you to hit the silk.

But more likely, management has been lied to, misled, bamboozled, and spun so many times that they have come to believe that nothing works but pressure.

Give the process a chance. Most managers really are more than ready to use accurate information to make better decisions. Every time you do this process, your knowledge of what will happen improves. If every time you report, you tell them what you really believe will happen, even the most dull management will finally figure out that your estimates are better than whatever they are shouting.

In January you’ll say: “Here is our schedule, it shows we’ll be done in September”. They’ll shout at you that it must be done in June. You say, “We’ll try, but if this is what we have to do, it’ll be September.”

In March, you’ll say: “Notice that we got done about what we predicted. Our schedule shows that we’ll be done in September.” They’ll shout, but they won’t be so sure. Expect them to say you have to work harder. “We’re working as effectively as we can,” you reply. “Would you like to see our Tracking Report?”

In May, in July, your schedule shows you converging on the predicted date. Even if the date shifts, it will shift less and less each time you go in. It must: there’s less to do each time, so your error would decrease even if you weren’t getting better at estimating. And you are getting better.

Even the most pointy-haired management will get it. And if they don’t, you’re still the best-estimating, most effective team in the company. How can this be bad?