Faithful readers know that I’ve been moving XProgramming.com to RonJeffries.com, with the help of Bill Tozier, since September last year. Much has been learned, some of it good. Here’s a bit of a summing up.

Top-Level Requirements Oddly Static

The initial requirements looked like this:

When Laura moves on, I’d like to be able to maintain the site’s structure as well as content.

I’d like it to be easier to write articles: I’m using Markdown a lot lately and liking it. It’d be nice to just write an article, click a button, and voila! it’s up on the site, pictures and all.

I’d like to build something made out of software, and to learn a bit in the process.

Those requirements mostly increased in number. The basic requirements stuck. We learned quite a lot.

Technology Changed

We were originally making the site run live, rendering from Markdown via Ruby/Sinatra.

It became clear that since my “real” requirement was a low-maintenance way of just writing articles, having a requirement for live Ruby scripting on my ISP was not the way to go. I say “clear”. I mean clear to me after a while.

So we changed direction to creating a static site, and switched our tool-set to Ruby/Jekyll/Liquid, in November. That decision has stuck. The good news is that I can run on any server that can serve html. The bad news is that I will have a technology stack on my computer, and that will require some maintenance. We’ll address that further below.

Documentation Changed

My original plan had been to document each iteration’s work in detail, as I have done with other projects in the past. That idea quickly fell by the wayside. This is the 18th article in the rj-com series. One might have expected around 30 if they were one a week, and of course many more if they tracked each day.

Fact was, with other things to do, such as finishing Nature and having a life, articles were hard to come by. In addition, documenting our fights with Jekyll and Liquid would have produced some colorful language but not much useful information. I do think there might be value to documenting some of our findings about plug-ins and such. I’ve done a little of that already in some of this category’s articles.

TDD Changed

We had every intention of TDDing everything. It is, after all, nearly an article of faith in the Agile line of work. We found it almost impossible to TDD against Jekyll and Liquid. We did TDD some of our larger chunks of work, but by and large, it just wasn’t on.

You may be inclined at this moment to say “How dare you promote TDD and then not do it.” What I actually do, however, is tell and show people what I do, especially including TDD, try to give them enough experience and sense of it so that they “get it”, and then leave them to decide when and how to use what they’ve learned. I get to do that too.

We got away with it.
(What people mean when they say the way they chose turned out to be best.)

And what I’ve learned is that TDDing into this complex and opaque framework was not feasible for us. We made progress without it, we used it when we could see how, and we got away with it.

Would we have gone faster had we taken the time it would have required to be able to TDD more deeply? There’s no way to know. There are some cases where we were stuck for a day or more on how to do something. TDD might have helped with that, maybe. There were cases where we displayed debugging information on a test web page. You know, like print statements. TDD might have made that a bit faster but it might have slowed it down.

And we have had few if any regressions. So TDD would not have provided much in the way of help.

Would we have done better with more TDD? I honestly don’t know. We made the best decision we could at the time and lived with the results. The results are good, so at least our decisions weren’t fatal. Honestly, I think at the end of any project, that’s the best we know: We did our best, and got away with what we did.

Estimation Remained Impossible

As I reported in Estimation this Effort is Still Hard, we were essentially unable to estimate anything. The main reason was “the mangle”, wherein we discover what the work is as we do the work. Let me give some examples.

RonJeffries.com

We realized soon that we needed a way to release incrementally. A separate test site was a possibility. Then we realized that I own RonJeffries.com so why not use it. In addition, in this day and age (as people from my day and age like to say), my named site might make a better identity for me anyway.

This realization started a subproject to get a place for the new site to stand, which took rather longer than we’d have liked, because I insisted that it be on my current ISP, who is a bit of a friend, and he was having personal difficulties holding things back. Soon enough, we had a place to deploy, but “soon enough” means a few elapsed weeks.

Meanwhile, this change focused our efforts on moving things to the site, and created lots of stories in our “scraper” project, the part of the effort that moved the WordPress data into a form we could use on the new site.

More and more conversion requirements arose as we did more and more conversion. Can’t estimate what you can’t foresee.

Scraper

The aforementioned “scraper” initially had the job of converting my WordPress-based export into the new site format. This included arranging things into individual folders, one for each article, which was the form I wanted for the new site. (It was also nicely compatible with WordPress’s quaint tricks with article slugs.) Scraper was initially supposed to convert HTML back to Markdown, for easier editing of past articles.

As we went forward with that, we ran into issues. First of all, Bill and I had conflicting goals for conversion to Markdown, which we had never really identified, and in fact sometimes one of us would do something intended to meet the other’s needs but in fact we had their needs wrong. Amusing, since we did most of our work sitting across a small table from each other.

We finally decided that unwinding the some-times horrible HTML from XProgramming was not a good idea. So we removed that requirement. When I edit an old article, I can edit the HTML or convert it manually to Markdown if I wish.

That change saved a lot of time from when we made it, but we had already invested in the conversion capability only to throw it away.

We couldn’t estimate how long it would take to convert and then decided not to do it.

Design and CSS

Hidden in the requirements “convert me off WordPress” was just one little thing: I wanted a completely new look and feel. That had been chatted about, but since I didn’t know what I wanted, we chanted “CSS” and waved our hands at it, kind of going “Booga Booga” in hopes it would go away.

Truth is, a large percentage of the work time on the site has gone into its look and feel, which I like even if you don’t.

We can’t estimate what we don’t know. We didn’t know what we wanted until we began to see it.

Jekyll/Liquid

Jekyll has some of the best-looking documentation on the Web for tools like this. That is not to say it is some of the best documentation: it may not be. There’s a lot left out in the Jekyll docs, especially if you don’t want to strictly follow their blog-focused style. And we didn’t. My site is articles. It has a lot of blog-like characteristics but is not a blog. We had to figure out how to re-implement Jekyll’s blog capabilities for a site made of pages, not posts.

We now have several plug-ins which we had to build:

  • cat-pager builds category indexes for each category and creates the index page for each one. There are two “Generators” involved in this. One creates a YAML file containing the index information, and another uses it to build pages. The index creator caches index information from the old site, because there are about 400 articles and it takes 15 minutes to build the whole thing.
  • kramdown-with-pygments plugs Krandown and the Pygments code highlighter into Jekyll for our use.
  • regardingCategoryBlock builds a Liquid plug-in that sets a page up to loop over a single category. It’s used in the category pages built by cat-pager.
  • pinnedArticleBlock allows me to put a “pin” number in an article, and then choose that article for the index page. Currently the Nature article on the front page is pinned.
  • make-classic-json builds a JSON file of the articles in category Classics, and stores it in the site, so that the front page can display a random article every time you visit, using some Javascript that reads that file and formats a live article teaser.

Some of these may have been predictable had we spent a long time generating requirements before we started. Doing that would have had some bad effects:

  • We would have delayed the start of the project. Since we did pretty much everything we thought of, the project itself would have been equally delayed.
  • We would likely have fallen in love with at least one stupid requirement. That would have made the project run longer still.
  • We would not likely have known that these ideas would require plug-ins: Jekyll has capabilities that nearly do what we needed, just not what we needed.
  • We might have compromised the site to use the framework more directly, that is, by setting a policy against plug-ins when it was too soon to know how badly they were needed.
  • We would still not have eliminated the surprises. The make-classic-json plug-in, for example, was discovered and implemented this week.

Expertise Would Not Have Helped Much

I recall that early on in the project I commented, in support of the #noEstimates idea, that there was really no way to estimate it, and it’s only a web site. Someone tweeted to me that I was a fool because I could just as this here guy, with a link to that there guy, who was a specialist in converting WordPress sites. That there guy, it turns out, was out of business.

Even if he had been in business, what I know now is that a vanilla conversion would have lost large quantities of the site, which had been converted into WordPress (twice) and contained a lot of material that was not in standard WordPress form. And, of course, a vanilla conversion wouldn’t have done what we wanted anyway.

Bottom line, I just did this project and I’m here to tell you, I couldn’t have estimated it at the beginning, and if you handed me another site needing conversion, I couldn’t estimate it much better now. It would depend too much on things neither you, nor I, could know going in. To find those things out would cost you a very significant fraction of the total price (on top of the total price for the doing), and the would still be quite wrong.

Learnings

Oh there’s a lot!

Pairing

I commented in an earlier article about Difficult Pairing. It still is. Bill and I seem almost never to have the same assumptions about the code or about how computers work. He’s always wanting to look up something, or try something, while I am on a roll and just need the usual give-and-take from one’s pair about how to say something. I’m always objecting to the direction he’s going when he’s typing.

It’s not about he’s wrong and I’m right, or the other way around. Sometimes he’s on the right track, sometimes I am. But we’re rarely on the same track. Sometimes that has value: diversity, y’know. Often it’s distracting: off-track, y’know. Always, it’s oddly frustrating in a we’re still friends kind of way.

Framework

I’ve never met a framework that I liked by the end of the project. Jekyll/Liquid is no exception to that. “All” we have to do here is:

  • Convert a new Markdown article to HTML. That takes one call to any converter, such as Kramdown.
  • Put the article in a suitable folder in the site. However we do that we have to do it. Subfolders in my rj-com folder work fine for that.
  • Find all the categories and generate an index page for each.
  • Build a front page based on recent articles, pinned articles, and the random Classic.
  • Render the pages into a handful of standard layouts for articles, indexes, and so on.

This seems so easy that I’m tempted to throw Jekyll away and do it all in Ruby. Of course I’d have to invent a way to create the category info, and to cache it, and to skip generating the old articles, and to build the JSON and … oh a whole list of “little” things.

I kind of wish we had built all this in our own Ruby framework, using available libraries. Would it have taken less time? I think not. Would it have wound up a better design?

Of course I think it would be a better design, because I can see now what a better design would be like. Even seeing that, a half-century of programmng experience tells me I’d not be likely to get anything that good, no matter how smart I am or how hard I might try.

But I don’t like the framework. I don’t hate it. But I don’t like it.

Language

I like Ruby and it didn’t take long to get back into it. But everyone knows that the right way to do this stuff today is with Haskell or Ember or straight Javascript or Node or [add your favorite right here].

Ruby was a low-pain way to go and works just fine. Might have learned more going another way. Did I mention Clojure?

TDD

At this writing, I don’t regret the amount of TDD we did. Most of our work was tiny little bites, which tend not to need much TDD attention. We did get some good results, and much better understanding, when we tried to TDD.

I think the balance was about right, given a two-person project, but I’m going to push harder on TDDing whatever I do next.

Estimation

I see no useful way to have done more estimation on this project. It would not have changed anything we did that I can see. The only possible result of estimation would have been to decide me not to do the project, and I don’t think that would have happened. If it did happen, I’d not get the benefit that I have today of a cleaner site and an easier writing mechanism.

Innumerable Details

We’ve learned a lot about living with Ruby, Jekyll, Liquid, Kramdown, and even a bit of Javascript. We’ll look at whether documenting some of this would be helpful to some part of the community, to share a bit of that learning.

TL;DR

RonJeffries.com is alive and well, and still improving. We learned a lot, most of it good. We still couldn’t have estimated it. Welcome!