Bill and I met Thursday. My plan, hatched before he arrived, was that we might install Jekyll in a test folder and do some end-to-end testing, which we’d use to drive development of the real code, using the snippets from the existing tests as a basis. So I created the “real” Dropbox in box folder, which I called _articles, since the target folder on the remote site is called articles. I populated the folder with a couple of sub-folders for aggregating articles, and put in a few dummy articles. It looks like this:

  • _articles
    • 017-02ff
      • test-a
        • index.md
      • test-b
        • index.md
    • 017-test
      • test-c
        • index.md

That looks just like the shape of my main articles folder and in fact 017-02ff is the real name of one of my folders. Using that will let us confirm that the code that moves articles from the “in box” into the Jekyll site is non-destructive. I just got that in place when Tozier arrived.

We touched base and decided first to review our old sketch test for the overall app. I’ll put it just below. I’d suggest you glance at it and then move on. It’s not worth understanding as it’s just a talking point between us.

require "minitest/autorun"
require "net/ftp"
require "./passwords"
require "set"

class Test_JekyllRunner  < Minitest::Test

  def init_folders
    FileUtils.rm_rf("./_mac_articles")
    FileUtils.mkdir("./_mac_articles")
    FileUtils.mkdir_p("./_mac_articles/articles/article-3")
    FileUtils.touch("./_mac_articles/articles/article-3/index.md")
  end
    
  def test_elementary_flow
    # sync the files
    # run jekyll
    # if it worked
      # ftp the files
      # commit and push
    # is everything in ipad_articles also in mac_articles?
    # is everything in mac_articles also in _site/articles?
    # is everything in _site/articles also on ronjeffries.com?
    # ipad = runner.getIpadArticles
    init_folders
    runner = JekyllRunner.new
    runner.sync
    Dir.chdir("_ipad_articles")
    from_directory = Dir.glob("**/*")
    Dir.chdir("../_mac_articles")
    to_directory   = Dir.glob("**/*")
    diff = from_directory - to_directory
    assert(from_directory.to_set.subset?(to_directory.to_set), "missing: #{diff.inspect}")
    assert(File.exists?("articles/article-3/index.md"), "did not preserve old contents")
  end
end

class JekyllRunner
  def sync
    FileUtils.cp_r("_ipad_articles/.","_mac_articles")
  end
end

Bill and I looked at this and looked at the folder structure I had built and decided that no, we weren’t ready to start with end to end. We wanted to beef up our FTP test. Last time, it looked like this:

require "minitest/autorun"
require "net/ftp"
require "./passwords"

class Test_FTP  < Minitest::Test

  def setup
    FileUtils.rm_rf("./_target")
    FileUtils.mkdir("./_target")
  end

  def test_target_is_empty
    target = Dir.glob('_target/*') 
    assert_equal(0,target.length)
  end

  def test_connect_finds_no_files
    ftp = ftp_connected_to_target
    list = ftp.list('*')
    ftp.close
    assert_equal(0,list.length)
  end

  def test_upload_files
    ftp = ftp_connected_to_target
    oldDir = Dir.pwd
    Dir.chdir("/Users/ron/Dropbox/_source")
    source = Dir.glob('**/*').sort
    source.each do |file_name|
      if File::directory? file_name
        ftp.mkdir file_name
      else
        ftp.puttextfile(file_name,file_name)
      end
    end
    ftp.close
    Dir.chdir(oldDir)
    Dir.chdir("_target")
    target = Dir.glob('**/*').sort
    assert_equal(source, target)
    Dir.chdir(oldDir)
  end

  def test_make_subfolder
    ftp = ftp_connected_to_target
    ftp.mkdir("subfolder")
    ftp.mkdir("otherfolder")
    ftp.mkdir("otherfolder/subsubfolder")
    ftp.puttextfile("test-ftp.rb","otherfolder/subsubfolder/RUBY.rb")
    ftp.close
    exists = Dir.exist?("_target/subfolder")
    assert(exists,"folder not made")
    exists = Dir.exist?("_target/otherfolder/subsubfolder")
    assert(exists,"subfolder not made")
  end
    
  def ftp_connected_to_target
    ftp = Net::FTP.new('localhost')
    ftp.login(Passwords::USER,Passwords::PASSWORD)
    ftp.chdir('programming/test_ftp/_target')
    ftp
  end
end

Our concerns had to do with copying down the in box structure, and copying the output structure up via FTP. Same problem, really, using different tools. The fundamental issue, either way, is that we need to create a series of nested folders, copy contained files into them, non-destructively.

Given a more realistic picture of what we wanted, based on the structure I built while waiting for Bill, we had a better picture of what had to be done. Here’s our central test to talk about:

  def test_make_subfolders_safely
    ftp = ftp_connected_to_target
    ftp_mkdir_safely(ftp, "subfolder")
    ftp_mkdir_safely(ftp, "subfolder")
    ftp_mkdir_safely(ftp, "otherfolder")
    ftp_mkdir_safely(ftp, "otherfolder")
    ftp_mkdir_safely(ftp, "otherfolder/subsubfolder")
    ftp_mkdir_safely(ftp, "otherfolder/subsubfolder")
    ftp.putbinaryfile("test-ftp.rb","otherfolder/subsubfolder/RUBY.rb")
    ftp.close
    exists = Dir.exist?("_target/subfolder")
    assert(exists,"folder not made")
    exists = Dir.exist?("_target/otherfolder/subsubfolder")
    assert(exists,"subfolder not made")
  end

This test is evolved a bit from what you’ve seen before. We’ve broken out a new function, ftp_mkdir_safely, whose job it is, well, to make a folder if it doesn’t already exist. The test does all those folders twice, to be sure it works. When that function doesn’t work, either it tries to make a folder a second time, or it fails to make it at all. We wrote this in-line, as is our chosen style following Keith Braithwaite, and then extracted some functions:

  def ftp_mkdir_safely(ftp, folder_string)
    ftp.mkdir(folder_string) unless 
        ftp_folder_exists?(ftp, folder_string)
  end

  def ftp_folder_exists?(ftp, folder_string)
    split = folder_string.split("/")
    proposed_folder = split.last
    ppl = split[0...-1]
    proposed_prefix = "./" + ppl.join("/")
    ftp.list(proposed_prefix).any?
        { |name| name.match(proposed_folder) }
  end

This is better than we had, for two significant reasons. First, it handles the actual case, rather than some guessed-at sample cases. Second, the code is coming to be a bit better (in my opinion, which at this computer shall prevail). It’s a bit more expressive, and begins to implement ideas that we have in our heads, such as “create this folder unless it already exists”.

All That Jazz

I’m reminded of a scene in All That Jazz, where Joe Gideon has just made yet another cut of the film, while being hassled by producer Joshua Penn to get done. Gideon pays no attention to Josh, and when he finishes the cut, Penn says, ruefully, “It is better. Oh God, it is better”. How do we trade off better against done?

Here within these walls, we’re interested in learning and in discovering errors. We have no deadline. And we’re really pretty ignorant about what we’re doing, so I’m happy to learn. On the other hand, I’ve had projects canceled under me or pulled away and given to someone else, when I didn’t deliver on an expected date. The general guidelines I recommend are to have the product always ready to ship, with the most important features, as seen by the business people, in it and working. This creates a much healthier dynamic, where the business people really do have control over the show and can ship it whenever they want.

In the All That Jazz scene, if the film had been ready to go, and Joshua had control over whether to ship it, things might have gone better, though it wouldn’t have made for such a good scene. And if it had been Joshua asking to have the scene improved, it would have been even better from a business viewpoint, though of course Joe Gideon would have been ticked off to be sent back to the editing room.

One point is, though, that what makes a great movie makes for tensions between business and development that can be managed better than they often are. And the point here is that we’re here to learn and we’ve learned something valuable, so our couple of hours are well spent in my opinion.

The Mangle

Andrew Pickering has written The Mangle of Practice, in which he argues that, in science, the things we work with act like active agents, shaping the work as much as we do.

In his view, machines, instruments, facts, theories, conceptual and mathematical structures, disciplined practices, and human beings are in constantly shifting relationships with one another—”mangled” together in unforeseeable ways that are shaped by the contingencies of culture, time, and place.1

Many of us in software development have the same feeling about our work. I think it was from Kent Beck that I first heard the idea that “the code tells us” how it “wants” to be designed. I can assure you that often it feels to me as if the code is telling me things. What’s really happening? I suppose it’s that I’ve become sensitive to how easy or difficult it is to do something, and when it gets more difficult I interpret that as the code pushing back against my bad idea.

One of the sections of The Mangle in Practice (note “in” not “of”), by my colleague and friend Brian Marick, expresses this feeling particularly well, and consistently with Pickering’s notions. Frankly, I found Brian’s article far more readable than Pickering. Shorter, too.

A Better Mangle?

I started the work reported in this article with great confidence that we’d have at least one end-to-end almost-releasable version of a half-decent test of the real thing by the end of the session.

The problem, and the code, had their own ideas. Their idea was that we didn’t yet know what we wanted, and we didn’t yet have sketched code that did what we wanted. The problem and the code pushed back against my cunning plan, and we listened and went with what they wanted.

Had we not listened, we might have forced out an end-to-end test, or we might have failed utterly. We might have come up with an improved abstraction, or we might have built a kludge that would ultimately be harder to make correct, more difficult to make well-designed.

My preference is to listen to the problem and the code, and to pay attention to what they’re telling me. Sometimes, I’ll decide to go with what they’re telling me and improve the design. Sometimes, I’ll take the risk of pushing on to release over their objections. When I do the latter, I watch for trouble down the line. Almost always, I come out believing that listening to the problem and code’s suggestions would have made things go better. Of course, that’s hindsight, and we all look back on our previous work and think we could have done better.

I’ll not try to tell you how to make those decisions. I do advise, however, that you try to learn from my “erors” here, and from your own, so as to fine tune your sense of when to make another cut of the film before releasing it. But do, if you possibly can, have a version ready to go in case “they” would prefer to release it as is.

See you next time!

  1. From the University of Chicago Press book blurb. ↩