Extreme (Programming) Thoughts

I’m taking a couple of on-line comments as writing prompts today. Let’s see what we get.

Last Tuesday night, Chet said something to the effect that when we were first talking and writing about Extreme Programming, we said that it was simple enough that ordinary programmers could do it. He went on to say that on the C3 team that first did XP, the team was self-selected from a group that had been immersed in the domain, and in Smalltalk, for well over a year, and that had all been programming for quite a few years before that. He wondered whether we should have said that yes, you do have to be good to do XP.

Tim Ottinger found a link, and some history, where the XP values had “aggressiveness” rather than “courage” (and, later, also “respect”). I am not sure but I think that when Kent first introduced C3 to the ideas, he did say that we would be “aggressive”. He didn’t mean with each other, he meant as regards going after good things with some fervor.

In subsequent threads, Martijn Meijering said that he felt it connoted “drive” or “striving for excellence”. He went on to remind us that we used to say merciless refactoring, relentless testing, continuous integration, the simplest thing that could possibly work, and to suggest that it was the italicized extreme bits that seem missing from today’s teams.

Those are today’s writing prompts.

Extreme Programming, Today, for Me

I emphasize today, and for me. Kent Beck assembled Extreme Programming, from his own experience and mind, and I claim no ownership of it, but will accept responsibility for mistakes I’ve made in trying to explain what I understood of his ideas, and what I understood from my own experience in trying what I understood of what I heard of those ideas.

The long indirection is important. Kent created the ideas, over time, from his life experience. I heard some of what he said and read some of what he wrote. In his speech and writings, he surely expressed his ideas imperfectly. I didn’t hear it all, and what I heard I surely often misheard or misunderstood. I integrated that with my own spotty experience and then said or wrote what I said or wrote, again a poor image of whatever was in my mind.

It is always thus.

Again, today, I’ll give you a poor image of what’s in my head and you’ll get part of it. We just have to hope that if we try often enough, you’ll find good things. They might not even be the good things I had in mind. That’s OK. Serving as a bad example is better than being unseen and unheard. I guess.

The Increment

My current/recent thinking centers around the “increment”, the running tested software which I believe a team should¹ produce as soon as possible² and keep running thereafter.

Look at that right there. I’m asking that the product grow in running tested ready-to-ship code from the very beginning, every day, until we finally stop funding the effort.

That [DELETED] ain’t easy!

Oh, if you’re really careful and somewhat skilled, you can do the first increment. It does have to be running and tested, but maybe it just displays a window saying

Caveat Emptor  
Please Log In:

It’s in the second and subsequent weeks that it starts to get difficult.

Testing

We can’t just test manually. Since we are changing the program every week, we need to test pretty much everything every week. So if it only took ten percent of a week to test what we write each week, three months in we’d be testing more than 100 percent of the time. We’d have to hire more and more testers, and presumably we have a fixed budget, so we’d have to lay off programmers, and pretty soon all we have left is one programmer and nine testers who can’t keep up with the load.

We need testable code, and we clearly need automated tests. (We need other forms of testing as well, but the bulk of our testing is retesting and we need that to be automated.) To do this requires at least two substantial skills:

First, we have to learn how to write code that is amenable to testing with automated tests. Second, we have to learn how to write automated tests. And third³, we have to learn how to write tests that don’t require rewriting every time something changes.

That [DELETED] ain’t easy!

Evolutionary Design

The design for “Caveat Emptor, Please Log In” is pretty simple, but as we add new capabilities, we need a bigger and better design. Since we need a new increment every week, we need to create the bigger and better design incrementally, week by week.

We call this evolutionary design, and it is not easy. To do evolutionary design requires at least two substantial skills:

First, we have to be capable of creating good designs. Second, we need to be able to move from one good design to a better one smoothly, so that we can do it every week. Third⁴, we have to do it without breaking anything.

That [DELETED] ain’t easy!

Refactoring

The main body of technical knowledge supporting incremental design is called “refactoring”, the practice of improving the design of existing code without breaking it.

Some refactoring steps are quite simple and easy. But the transition from one design to another isn’t always simple and easy. Sometimes we just can’t see a series of simple steps to get from point A to point B. Sometimes, even if we can see them, there are so many steps that the code would remain broken for a long time. We don’t have a long time. We need to deliver another Increment, this week, if not tomorrow. Ideally tomorrow.

That [DELETED] ain’t easy!

Test-Driven Development

If the need to test everything and refactor all the time were not enough, the best approach we know today to that is “Test-Driven Development”, the practice of writing a very simple test showing that one tiny bit of the product is missing, and then coding that one tiny bit, then refactoring to be sure the code is good, and then repeating.

What experienced practitioners discover is that the steps taken with TDD need to be small. Tiny. No, smaller than that. We often find that with our very smartest thinking cap on, the steps we take are too large. We rarely, every, find a step that is too small, because if it were too small, writing the test would be small and making it run would be small, so that the time lost is small. Smaller seems always to be better.

Believe me, someone who has been doing this for a quarter of a century now, TDD micro-testing is hard. The learning is endless, the need to refocus on smaller is endless.

That [DELETED] ain’t easy!

Frequent Integration

If we are a team of multiple developers working other than as an ensemble, we find that each subgroup needs to integrate its work incredibly frequently. It works like this:

If you and I have checked out the code and implemented some new capability, when we go to check it back in, clearly we have to be responsible for making sure it works. Our tests, and all the tests that already exist must work.

But if some other subgroup has checked in before us, our changes may not be compatible with theirs, and we’ll get a merge conflict or some tests will stop running. We might have to ask for help, which would be embarrassing, but one way or another, we own the problem of making all the tests run again.

But there’s a trick! When we check out, and do our work, our work clearly runs on the then-current head of the repo. So if we check in before anyone else checks in, our work will integrate properly! All we have to do is check out, work, check in, before anyone else checks in.

The only problem is that this trick works for everyone. So we all want to check in essentially immediately after checking out. I’ve seen two pairs of programmers racing to be the first to check in. That’s not often ideal: rushing is rarely a good software strategy.

Even so, the ideal thing is to work in very small steps, checking in after each time the tests run. It even turns out that if everyone works that way, the chances of a conflict go down, and even if there is a conflict, it will be small, because the changes that went before us were small.

I can’t prove this but my math intuition tells me that in the limit, zero time between checkout and checkin is ideal. We need to learn to do everything in tiny steps:

1. Check out;
2. Tiny test, doesn’t run;
3. Tiny code, test runs;
4. Check in;
5. Repeat.

Oops. Where’s the refactoring? Sooner or later, in one of those cycles, we notice that the design has deviated from good enough and we’ll have a longer cycle as we refactor.

Ideally, we’ll do this:

1. Check out;
2. Tiny refactoring;
3. Tests still run;
4. Check in;
5. Repeat.

That does get tedious and often we’ll find that we want to do quite a few refactoring steps before checking in. There’s a risk that we’ll have an integration issue, but the risk is small, because we’re refactoring and not changing how things work, so that most of the other subgroup’s changes will not break anything. Once in a while, they do, and we just deal with it.

That [DELETED] ain’t easy!

Extreme May Be Right

We’ve probably only scratched the surface of what we have to do well in order to create an Increment and keep it good. We have to learn how to do powerful automated testing⁵, incremental design, test-driven development, and refactoring. We have to do it all in very tiny steps.

There are other skills and practices that are part of Extreme Programming, or part of the general knowledge of good ways to do this stuff, and I’ll just give them a line or two:

Pair Programming

We need everyone to be able to work on everything, to the degree possible. We need to make very few mistakes. Accordingly we may want to work together in pairs, frequently switching.

Ensemble Programming

Also called “Mob Programming”, this is an approach where the whole team works together at one machine to build everything. It certainly resolves the continuous integration issue and practitioners often love the technique.

Whole Team

To move this quickly, we need the team to have all the skills necessary to build the increment. Not just coding and testing, but also the skill to decide what to do next. All the people with these varied skills need to work together effectively. Human skills are the hardest skills, and we need them quite urgently.⁶

High Skill

In the end, to write good software requires truly skilled people. People with little experience will have lower skills, by and large, than folx with long experience. This means that if we’re to do this well, we need to increase people’s skills as rapidly as we can. (And, of course, where possible, hire people who already have the skills.)

The fact is, the skills to work in the iterative incremental fashion called for by Extreme Programming, and its more famous⁷ relatives Scrum and SAFe, are not built in to programmers off the street. They are not taught in most schools. They are not present in the examples you find in most books, or on Stack Overflow or other online sources. They’re out there, but still not generally known.

Bottom Line

I’m calling it. Extreme Programming, and its relatives Scrum and SAFe, are very difficult to do well. They require the application of many skills with great teamwork. The skills are not commonly⁸ found in schools, in books, nor on line.

This is not a limitation of Extreme Programming as compared to Scrum or SAFe. The creator of Scrum said that XP practices are necessary to high quality Scrum. The skills are needed because of the fundamental idea of all these “agile” methods, the creation of a continuously improving increment of working software.

The skills are required in order to follow the Agile Manifesto. Continuous delivery of working software is hard, difficult, challenging, demanding, and not easy.

Software development is hard. The best ways we currently know to do it, the so-called “Agile” approaches, are hard. The not-so-good ways, often clumped under “waterfall” are, frankly, harder. The not-so-good ways tend to lose the essential property of software development, the development of software, reducing it to some kind of transcription of an ideal design. I’ve been doing software for six decades, and I can tell you, that trick never⁹ works.

Maybe we need to put the Extreme back into Extreme Programming. Maybe we need to put the Extreme back into all programming. Maybe we need to put the Extreme back into our lives.

It’s hard. It’s all hard. Maybe everything we do deserves an aggressive, relentless, merciless, continuous approach to performing with excellence.

Footnotes: