It's Alive!

Yesterday morning, Tozier and I tweaked a last few things and identified one more issue to resolve before moving all the XProgramming.com content to RonJeffries.com. Yesterday afternoon, I resolved that issue and pushed the content. A couple of tweaks seemed worth doing. It’s alive. Here are a few thoughts prior to our project retrospective.

Estimates

This project got started around September 20th last year. Elapsed time is nigh on to five months! I’ve written here a number of times about the difficulty of estimating when we’d be done. I had in mind that it’d be the end of the year. I’m not displeased with this result.

Much of the elapsed time is due to the fact that there were about nine hours of programming per week in the good weeks. So, at a guess, 200 pair hours, 5 pair weeks. That’s not bad. Of course, at $50 an hour for programmers, that’s about $10,000 for a web site, but on the other hand it’s only about $25 per page, which isn’t so bad.

I’m pleased with the result, and feel that the time was well spent, even though there were many mistakes, and much learning.

I am now one of the two world experts at converting a complex WordPress site to a static site, using Ruby, Jekyll, and Nokogiri. If you told me you had a 500 page WordPress site to convert and asked me to convert it, I should be able to give a really good estimate.

Well, we’ve got these two tools, Scraper and the Jekyll setup I call rj-com. So we’d just export your site to XML, run it through Scraper to get it broken out into articles, and push those through rj-com and voila! your site would be done. We should be able to do that in two hours. Add in a week to clean up the inevitable loose ends.

So, a week, right? Let me think about that while I write on some other topics.

Requirements

The big picture was easy. People in Twitter kept telling me that any expert (there used to be none, now there are two) could quickly tell me what it would cost to convert my site. I knew it wasn’t that easy but I wasn’t close to understanding the requirements. Most of them were discovered along the way. One of them was discovered yesterday. Here are some that popped up.

• Design a new look and feel
• Build that out in CSS and content
• Render the site quickly, from Markdown, on the server
• No, that’s still too much on the server, go full static
• Make it easy to write a new article, assets in a folder with text
• Convert old articles to Markdown to make updates easier
• No, that’s getting too hairy and we’ll not get it right. Drop that.
• Create indexes including old and new articles
• Takes way too long to generate the full site make it faster
• Takes too long to upload the site, make that faster
• OMG there are important pages on XProgramming that are not in WordPress

It went on and on. As I mentioned in Difficult Pairings, I think this is what is called “The Mangle of Practice”. Doing the work discovers the work.

There was no way to estimate this project because we didn’t know the requirements. There was no way to know the requirements: we had to discover them by doing the work.

Mistakes

Along the way, we made many mistakes. If we had an old style manager, he’d have been shouting at us to work smarter. (I am reminded of Chet Hendrickson’s example of really bad advice: “Be taller”.) In addition to all the little mistakes (see “Tests” below) we made some big ones, including:

Sinatra

The original plan was that the site would be in Markdown, all in nifty folders on my computer, and we’d just push up new articles and render them in Sinatra.

But the point of this was to reduce the need to update weird software on the server and our experience in setting up a server led me to decide to go full static. This resulted in a shift to Jekyll.

Scraper

Scraper is the tool that pulls articles out of the XML export of XProgramming, and arranges them into the input folders for Jekyll. Tozier has a somewhat different set of needs for this, so we wound up putting options and switches into it. It became an obvious bottleneck. See also “Tests”.

We realized we had a huge backlog of “necessary” items in Scraper and struggled with keeping those understood. I finally expressed the backlog as some RSpec tests, which helped us sort necessary from nice and also got us some tests.

Scraper had grown by accretion and was pretty ugly. We finally started over and wrote a couple of classes, WpPost and WpPostFactory, to contain the work and do it. Once we did that, things went much better.

In the end we made Scraper better by removing capability and writing better code.

Inventions and Discoveries

We had to discover all the tools and gems. We had to choose among them, which was mostly easy because Tozier had some experience and preferences and usually I didn’t care.

Then we had to learn to use them. In particular, Jekyll doesn’t really want to build a web site: it wants to build a blog. Much of its useful capability is all entangled with the particular blog style they’ve chosen so we had to invent ways of building indexes and the like.

Jekyll’s documentation is deceiving. It is perhaps the most attractive documentation of any of the tools out there. It is well organized, and comprehensive. And it is nearly useless. In particular, it tells you what you can do but not what will actually happen nor how to do what you want. You get a real love/hate relationship with the documents.

Searching for help with Jekyll, you find the same issues again and again, quite often without resolutions. In the end, by water-boarding Google, you can get answers which may or may not be true.

Jekyll’s rendering engine is Liquid. It’s good when it’s good and when it’s bad it’s horrid. Liquid has a scripting language designed to be safe in the hands of fools and malefactors, which means you can’t do half the stuff you need to without kissing your own elbow.

Jekyll and Liquid share notions of the site, its pages, and the attributes of the pages. If there’s a way to know just what those attributes are and how they’re set, it’s well hidden. We finally figured out only this week that Liquid really runs on Hashes. Well, Hashes with some kind of override built in that allow you to say page.title rather than page['title']. Sometimes.

Mind you, it’s all good enough: the site is up after all. But we really had to jump through our own orifices a few times to get things done. We wrote plugins for Jekyll and Liquid. This is possible but the documentation leaves much to be discovered. We spend rather a lot of time dumping hash tables in the middle of the run to see what was going on.

Suffice to say, the tools were like most tools we programmers have. Invaluable but incredibly frustrating.

Tests

You’re probably aware that I’m all for tests at a unit and acceptance level, or what I prefer to call programmer and customer level. We wanted to have tests, we really did.

The problem is, Nokogiri and Jekyll amount to legacy code. If they have tests, we don’t know where they are and we want to test them in use, anyway, not see if they work. So how do you write a test to see whether you can get the right information out of a Jekyll page (or a Liquid page, which isn’t quite the same thing)?

No problem, first get an page that has in it what you want. Wait, what? Pages have no accessible creation method, they just come into being as Jekyll runs around in your site folders.

When we were creating our own objects, our Scraper WpPost, or our Jekyll Generators, we first wrote spikes and then some RSpec tests. But much of what we did is unsupported by tests because we didn’t know how to write them and we needed to get done.

Sound familiar? No time to test, we need to get done? Yeah, well, I have a greater appreciation for the power of that argument when you do know how to program and you don’t know how to write a test.

I think when we did write tests, they always paid off in quality and speed. But sometimes we could not bring ourselves to take the time off to figure out how to write them. I’d like to go back and look now, to see whether we have learned enough to do better next time. (There will be no next time.) If we do go back, I’ll report here, of course.

Bottom line, testing helped immensely and often seemed impossible to automate.

Pairing

We paired most of the time. I usually resisted working alone, and Bill mostly did spikes or worked on his own stuff when he was alone. Once in a while we built things separately, especially after we began to see how to do tests.

Pairing was painful as I reported in the already-mentioned Difficult Pairings, but it was very valuable. I think I’m a pretty good programmer (note the unassuming and fake modesty in the preceding) but all the pair programming good stuff happened when we worked together.

When one of us was stalled or confused, the other had an idea. We kept each other on our toes on code quality, despite some pretty serious differences on what good code looks like.

When it came to the tools, one of us could surf for ideas while the other one banged head on the keyboard. Bill’s deeper familiarity with the tools helped me use them. His absurd affection for RSpec made me give up and learn how to use it. While I don’t love it, I’ve certainly come to like it and respect it.

Just having someone to talk with who was inside the project and understood the requirements, possibilities, and code made a huge difference.

I’ve been doing much of my programming alone for quite a while, with toy things at home, or Codea, or Second Life scripting. I had gotten used to how it is and I’m pretty good at working that way (vide supra regarding pretty good).

Pairing is better. It’s less stressful, goes faster, and more gets done. I don’t know how I’ll find people to pair with but I bloody well should.

Focus

Bill and I are easily distracted and often would drift into discussing what we should do or how we should do it, rather than keeping our eye on the ball. Sometimes Chet would be at the BAR with us and we’d talk about random things. Other times he’d notice we were being unproductive and would try to “help” us. This lack of focus didn’t trouble us much: we’re friends, doing this because we want to, and we’re not on anybody’s clock. Were we being paid for this, we’d have to keep more focused.

I’m not sure that would have been better. It might have gotten us done sooner, but I suspect we’d have had much less understanding, much less learning, much worse code, and much less fun.

Incremental Release

The “Agile” style I promote calls for continuous incremental release. Often the people behind a project tell us that they can’t release anything until they have everything. We never believe them.

I couldn’t see how to release anything without everything at first. The plan was to replace XProgramming.com in place, with a static site. You either have the new site or you don’t. I couldn’t allow the site to suddenly lose most of its articles. No possibility of incremental release there.

Then I remembered that I own RonJeffries.com and decided to build the new site there. That let us experiment with the layout and such and also let me get a few articles up and begin drawing readers to the site. And I added “value” to RonJeffries.com by putting articles there that were not going onto XProgramming. So the site progressed.

But the big job was still there: get all the XProgramming articles moved over. I still see no way to release that incrementally. As soon as our automation makes one article look OK, it will make (almost) all of them look OK. Before that, you have no article you’d like to put up. One day you have none, the next day you have them all. And there’s no value to the old ones anyway, until we can redirect XProgramming to point here.

We were working incrementally without releasing. We improved code (and tests) daily, we committed code multiple times per day, and things were improving. When the look and feel was improved, we released it immediately. When the old articles were improved, we knew it, but they were still not suitable for release.

Until yesterday. So I shipped it. Woot!

Estimates, Revisited

So. A week to convert the next one, working full time?

Not on your life! We can convert one just like XProgramming in that amount of time, if there are no surprises, if the formatting is the same as mine, if the assets are all as well organized as mine, if you want the folder layout like mine, the CSS like mine, everything like mine.

We could convert my site again in a week, perhaps less. Yours? Honestly, it would be better just to start and see how it goes. I’m pretty sure we can do a decent job for less than $10,000. Might be substantially less. Could possibly be more. I promise to tell you if it looks like going over.

I’m sure of this: it would cost more to get a reliable estimate than it would cost to do it.

Next Steps

Prediction is very difficult, especially if it’s about the future.
– Niels Bohr

What’s next? Mostly, I hope, I’ll turn my attention to new content. Meanwhile, Bill and I will be pairing on some things he’s working on. That will be interesting and we may write about that as well.

As for the XProgramming conversion, there are some things. There’s material on XProgramming that is not yet here, mostly material that isn’t in WordPress but just dangling there in folders. Some of the original XP material, for example. I’ll be bringing that over. There are some articles that didn’t format well: I may fix some of those. And there’s material that only a hoarder would preserve, and maybe I’ll remove some of that.

Looking more broadly, I’d like to get back to drawing. As you’ll have seen, my drawing skills leave much to be desired. There are conferences to attend. I’d like to get the guts back into Agile somehow.

And so on. We’ll see. Thanks for coming along, and keep those cards and letters coming in.

Oh, and please buy a copy of The Nature of Software Development, and a few extras for your friends.