We begin by putting some markdown files in our _source subfolders. We may wind up regretting this, because some bozo decided we would write this test to run on the real source folder. But this is part of the Erors™ series so it’s all good.

Having done this, we believe that running our test will create the site as before but that now we’ll have index.html files in there. We improve the test:

  def test_jekyll_run
    FileUtils.rm_rf("./test_jekyll_site/articles")
    ipad_folder = '/Users/ron/Dropbox/_source/.' # dot is key
    jekyll_folder = './test_jekyll_site'
    ftp = nil 
    jr = JekyllRunner.new(ipad_folder, jekyll_folder, ftp)
    jr.move_ipad_files # sorry, Demeter
    jr.run_jekyll # oops, Demeter, my bad
    assert(File.exist?('./test_jekyll_site/_site/articles/a.txt'), "can't find jekyllated a.txt")
    assert(File.exist?('./test_jekyll_site/_site/articles/subfolder/index.html'), "can't find jekyllated subfolder index")
  end

The test, as expected, creates the desired files. Our test checks a couple for existence. We also looked at the folder and see the others.

However, because this Jekyll setup is incomplete, we are not getting any category files created. For this test to really work end to end, we would have to put our templates and ruby snippets and such into our test folder.

There will now be a brief pause while we debate this locally. Take a break yourself, get a new coffee or something …

We don’t have a test for this. Our mission is to copy files from source to jekyll input, run jekyll, then ftp up all those, and only those files corresponding to the source folders, plus the category files. We figure this is a two-part process, copying the corresponding ones, and copying all the category files. Making our Jekyll more typical can wait for a failing test.

Now let’s work on writing a test for the FTP part, which implies saving and using the source structure information. This gives one to think.

  def test_we_know_what_to_ftp
    FileUtils.rm_rf("./test_jekyll_site/articles")
    ipad_folder = '/Users/ron/Dropbox/_source/.' # dot is key
    jekyll_folder = './test_jekyll_site'
    ftp = nil 
    jr = JekyllRunner.new(ipad_folder, jekyll_folder, ftp)
    jr.move_ipad_files # sorry, Demeter
    # jr.run_jekyll 
    assert_equal(expected_to_move, jr.what_to_move)
  end

This sketch assumes a new (private but testable) method on JekyllRunner, that can tell us what we want to know about what needs moving. I’m intentionally vague here because so far this is a sketch. We’re not clear whether we’ll have to call the Runner or whether it can answer this question, somehow, right now. Hang on, we’ll work it out. (Yes, this is really how I do this. YMMV.)

Our proposed implementation is:

  def what_to_move
    Dir.glob(@ipad + '/*/**')
  end

This created an interesting cascade of problems. First we just used our @ipad member variable in the glob. That basically returned itself, because in glob we need the */** trick. Appending that as shown seemed to be the right thing, so we accepted that we’d need to remove the trailing '. in our setup. We also observed that we do that about four times in our tests. This is called “duplication” and it’s bad. But we’re here to drain the swamp. Our test now looks like:

  def test_we_know_what_to_ftp
    FileUtils.rm_rf("./test_jekyll_site/articles")
    ipad_folder = '/Users/ron/Dropbox/_source'
    jekyll_folder = './test_jekyll_site'
    ftp = nil 
    jr = JekyllRunner.new(ipad_folder, jekyll_folder, ftp)
    jr.move_ipad_files # sorry, Demeter
    # jr.run_jekyll 
    expected_to_move = []
    assert_equal(expected_to_move, jr.what_to_move)
  end

This now fails for the intended reason, namely that we have not initialized expected_to_move. What’s irritating is that the only sensible way to initialize it is with a similar call to glob. That’s not testing anything of note. Another possibility is to look into the returned collection for a few key items.

Tozier claims to have a thought. He suggests that this method could return a better-prepared list, such as one that refers to the Jekyll-created folders. Something like removing the Users/ron/Dropbox/_source and prefixing _site/articles.

We interrupt this program with another Eror™. It should be **/*, not */** because we need the folder names. We now return you to your regular program.

What I think is “really” going on is that we have a low-level operation that ftps one thing, X, from _site/articles/X to ronjeffries.com/articles/X.

Our learning implementation positioned the input on the _source folder and the output on the articles folder, so that these long string prefixes didn’t occur. We could (and maybe should) do that in this test as well, assuming that we’ll position the glob and the ftp properly.

We are now confused. Therefore we shall press on. First thing, position our glob on top of the folders we care about:

  def what_to_move
    pwd = Dir.pwd
    Dir.chdir(@ipad)
    result = Dir.glob('**/*')
    Dir.chdir(pwd)
    result
  end

This returns an undecorated list of files and folders in @ipad, namely _source: ["a.txt", "anotherfolder", "anotherfolder/f5.txt", "anotherfolder/index.md", "b.txt", "pic.JPG", "subfolder", "subfolder/index.md", "subfolder/sf1.txt", "subfolder/st2.txt"].

However, here’s my concern: we take this list, preposition our cursors, prepend some information, move from _site/X to ronjeffries.com/articles/X. and everything is correctly moved. We know we can do it because we have similar code in our other test that does it.

To me this code is calling out for an object. A collection of, them actually. Maybe they are named FTP_transfer or something. I’m not convinced I know enough to build this object now. Maybe we can push the problem down further into our JekyllRunner and deal with it later on the basis of removing ugliness or something. Or maybe it’ll just not bother us after it’s done.

Nonetheless, the fact that these strings are just strings troubles us. There’s a lot of hidden behavior going on here, due to the fact that we secretly position the Dir folder and the Ftp folder so that we can just use the names. (Except don’t forget we have to change .md to .html.) When we look at all these Ruby objects really want us to work with file names as strings.

I was taught that manipulating strings is not communicative and that there is usually an object waiting to be born. So there’s this tension.

Still the big fool says to press on. We’ll see how clean it turns out and decide later, not because we’re lazy (though we are) and not because we’re in a rush (though maybe we are) but because we do not see, yet, just what kind of object is needed. Writing the code out longhand and looking at it then will tell us more. The risk of course is that we’ll say “oh hell good enough” and leave it. Stay tuned.

I suggest returning an array of ordered pairs, source_name, target_name, which will embody the renaming from .md to .html. That looks like this:

  def what_to_move
    pwd = Dir.pwd
    Dir.chdir(@ipad)
    files = Dir.glob('**/*')
    Dir.chdir(pwd)
    files.collect { |f| [f, rename(f)] }
  end

and results in this:

[["a.txt", "a.txt"], ["anotherfolder", "anotherfolder"], ["anotherfolder/f5.txt", "anotherfolder/f5.txt"], ["anotherfolder/index.md", "anotherfolder/index.html"], ["b.txt", "b.txt"], ["pic.JPG", "pic.JPG"], ["subfolder", "subfolder"], ["subfolder/index.md", "subfolder/index.html"], ["subfolder/sf1.txt", "subfolder/sf1.txt"], ["subfolder/st2.txt", "subfolder/st2.txt"]]

Of course if an array of strings is questionable, an array of arrays of strings is mega-questionable. I do like, however that this test gives me decent insight into what the runner will do when turned loose. So I am of at least two minds, and I don’t have any idea how many minds the piles of ants across from me have.

The ants have come through! This double array does tell me how the system decided to move anotherfolder/index.html, but we really don’t need to know that it magically converted that from index.md. So we can just return the converted name list:

  def what_to_move
    pwd = Dir.pwd
    Dir.chdir(@ipad)
    files = Dir.glob('**/*')
    Dir.chdir(pwd)
    files.collect { |f| rename(f) }
  end

  def rename(file_name)
    file_name.gsub(/\.md$/,".html")
  end

The resulting data is:

["a.txt", "anotherfolder", "anotherfolder/f5.txt", "anotherfolder/index.html", "b.txt", "pic.JPG", "subfolder", "subfolder/index.html", "subfolder/sf1.txt", "subfolder/st2.txt"]

We’re still not even testing for anything, since expected is blank. Tozier is willing to just paste this into the expected. I’m not deeply opposed to it, though I’d kind of like to have a more semantic test. I think we’ll just paste the result since it is what we wanted.

  def test_we_know_what_to_ftp
    FileUtils.rm_rf("./test_jekyll_site/articles")
    ipad_folder = '/Users/ron/Dropbox/_source'
    jekyll_folder = './test_jekyll_site'
    ftp = nil 
    jr = JekyllRunner.new(ipad_folder, jekyll_folder, ftp)
    jr.move_ipad_files # sorry, Demeter
    # jr.run_jekyll 
    expected_to_move = ["a.txt", "anotherfolder", 
      "anotherfolder/f5.txt", "anotherfolder/index.html", 
      "b.txt", "pic.JPG", "subfolder", "subfolder/index.html", 
      "subfolder/sf1.txt", "subfolder/st2.txt"]
    assert_equal(expected_to_move, jr.what_to_move)
  end

This test isn’t very robust but it does convince us that we’ll try to move the right things.

Now, we can do one of these things:

  1. stop
  2. write a more detailed test verifying what’s going to happen
  3. write the test that implements the actual move and check the results.

Tozier’s voting #2. I don’t know what the test would look like.

I have good news and I have bad news. We actually wrote a fairly clever test:

  # def test_what_will_be_ftped
  #   FileUtils.rm_rf("./test_jekyll_site/articles")
  #   ipad_folder = '/Users/ron/Dropbox/_source'
  #   jekyll_folder = './test_jekyll_site'
  #   ftp = nil 
  #   jr = JekyllRunner.new(ipad_folder, jekyll_folder, ftp)
  #   jr.move_ipad_files # sorry, Demeter
  #   moved = jr.move_files(simulated = true) 
  #   expected_to_move = []
  #   assert_equal(expected_to_move, moved)
  # end

The idea here is that the JekyllRunner will move files with an optional simulated mode, but will always return a list of what it did. We built that this way:

  def move_files(simulated = false)
    pwd = Dir.pwd
    Dir.chdir("#{@jekyll_folder}/_site/articles")
    # position Ftp
    ftp_connection = ftp_connected_to_target
    result = []
    files = what_to_move
    files.each do |file|
      result << move_file(ftp_connection, file, simulated)
    end
    result
  end

  def move_file(ftp, file_name, simulated)
    unless simulated
      # if File::directory? file_name
      #   ftp_mkdir_safely(ftp, file_name)
      # else
      #   ftp.putbinaryfile(file_name,file_name)
      # end
    end
    return "#{Dir.pwd}/#{file_name} to #{ftp.pwd}/#{file_name}"
  end

You’ll note that we didn’t trust our simulated flag and also commented out the move. Maybe this should have been the clue that the bite was too big.

This implementation produces a really nice list of incorrect from/to file paths. We tried several times to get things linked up right, realizing ever more strongly that we were in a rathole. The rathole probably is that the setup of this object needs better file/folder information, and that we have too many magical constant file names and folder names built in, and that we are still confusing the three stages of copy into jekyll, jekyll creates _site, object moves from _site to the ISP.

Our brains are full. We are still smart enough to get our tests back to running and hold this for another day.

It’s Erors™ all the way down. Possibly you are much smarter than I am and never get in this kind of trouble. Or possibly you do and never notice. Or possibly you do, notice, and never mention it because you assume no one else has these problems. My guess is that everyone has these problems and the point here is to make it OK to become confused and get yourself unconfused.

If that doesn’t work for you, then my point is to make you feel better about yourself because you’re much smarter than I am. All good.

Afterword

It’s Friday now. The above took place on Thursday. I’m at the BAR1 alone this morning and have some additional thoughts to share about this.

in general …

Tozier and I chatted briefly after closing out the above session. I said something similar to above, to the effect that either all programmers have similar occurrences to mine, where things just get sticky and weird and slow, and they’re confused but don’t notice or don’t admit it … or, I am uniquely bad at programming.

Bill said that he thinks that often, programmers are sitting heads down, banging on something, and they don’t quite understand it, and it’s slowing them down, and maybe they’re even starting to feel frustrated or stupid, but they don’t ask because they figure everyone else is really smart and they’re just … uniquely bad at programming.

One of the great things about pair programming is that while we may all be uniquely bad at programming, the word “uniquely” comes into play in a good way. Often you know the things I don’t, and the reverse, so that working together our unique badness cancels out a bit.

In a really smoothly-operating team, it’s even better. If both of us are confused, we can just raise our voice a bit and ask “does anyone remember how the MorgleMapper works?” and quickly get an answer or someone pops over to help. In a room where we don’t feel trust, we don’t ask because we’re afraid someone will say “Only a fool doesn’t totally understand MorgleMapper”.

So part of why I show my work as it really happens, full of confusion and errors, is to help you feel that it’s OK to be confused and to make mistakes, so long as you move quickly to become unconfused and to remove mistakes.

but this code …

We’ve been focusing on two things. First, we wrote some tests to learn how Ruby’s Ftp works, and how the Dir and File objects work, aimed at copying files and folders around. Second, we began to work toward an end-to-end test of our JekyllRunner object. All our experiments were very ad hoc, kind of hammering on the strings until we got what we wanted:

    oldDir = Dir.pwd
    Dir.chdir("/Users/ron/Dropbox/_source")
    # ten lines of other stuff
    Dir.chdir(oldDir)
    Dir.chdir("_target")
    target = Dir.glob('**/*').sort

Then we stopped, and while we had made it work, we hadn’t made it good, and we certainly hadn’t organized our thoughts.

In building the end-to-end, we looked mostly at the core functions, and didn’t look at the setup. So the setup is pretty weird. Here’s some of the code:

  def test_file_copy
    FileUtils.rm_rf("./test_jekyll_site/articles")
    ipad_folder = '/Users/ron/Dropbox/_source'
    jekyll_folder = './test_jekyll_site'
    ftp = nil 
    jr = JekyllRunner.new(ipad_folder, jekyll_folder, ftp)
    jr.move_ipad_files # sorry, Demeter
    assert(File.exist?('./test_jekyll_site/articles/a.txt'), "can't find a.txt")
  end

  def ftp_connected_to_target
    ftp = Net::FTP.new('localhost')
    ftp.login(Passwords::USER,Passwords::PASSWORD)
    ftp.chdir('programming/test-ftp/_target/articles')
    ftp
  end

  def move_file(ftp, file_name, simulated)
    unless simulated
      # if File::directory? file_name
      #   ftp_mkdir_safely(ftp, file_name)
      # else
      #   ftp.putbinaryfile(file_name,file_name)
      # end
    end
    return "#{Dir.pwd}/#{file_name} to #{ftp.pwd}/#{file_name}"
  end

  def rename(file_name)
    file_name.gsub(/\.md$/,".html")
  end

  def run_jekyll
    pwd = Dir.pwd # TODO fix this to be more reasonable
    Dir.chdir(@jekyll_folder)
    `jekyll build`
    Dir.chdir(pwd)
  end

Look at that crap! Look at it! It’s not that it’s wrong. Right now all our tests are running (since we removed the one that wasn’t). But the names are spread all around, and are not used in any consistent way. There are assumptions built in to Dir and Ftp about what folder you’re “in”, and sometimes we’re extending past that folder and sometimes we’ve set the folder to be “right”. We have things that need to change the working folder and then set it back, and they’re being down with ad hoc code that saves the previous folder and then, if we remember, restores it.

But wait, there’s more. We have repeated difficulty thinking about the three phases of this operation:

  1. We copy from a Dropbox folder to the input side of the Jekyll folders;
  2. We run Jekyll, which creates corresponding files in its _site folder;
  3. We FTP from the _site folder to a designated folder on the site.

The first and third need to use the “same names”, that is, we move a set of things into Jekyll input, and then, from Jekyll output, we FTP the things of corresponding names. Corresponding means “all exactly the same except .md files become .html files so we move those.”

The folders we process are processed differently. For the Dropbox folder we do a Dir.glob to decide what to move. For the Jekyll run, Jekyll magically creates a whole lot of stuff, including html copies of our input but a lot more. Then from Jekyll’s _site folder, we want to FTP the items that correspond to our input items. We’ll need to drive that from the saved glob, with some of the names changed to deal with what Jekyll does.

Don’t answer yet! You also get the fact that all the key operations are making assumptions about what folders are active. Where are we standing when we glob? Where are we standing when we jekyll build? Where are we standing when we do our Ftp loop?

Did I mention that I’m starting to think my decision to run the tests from the real folder may also be a problem, because there’s junk in there for the tests that I don’t want on the site? Well, I am starting to think that.

Arrgh! My life just isn’t working out! What’s going wrong??

Things are the way they are because they got that way.
-Jerry Weinberg

I’ve never found that Weinberg quote very satisfying, because I’m really inclined to look for causes so that I can fix things. But it’s really clear here, isn’t it? This code has odd bits of rot in it just because it got that way. We made little decisions, to put something off, to copy something that worked, to take a swag at a solution. We tested a lot but maybe we didn’t test the right things. We said things and thought thoughts and they added up to where we are now.

I’m not going to blame myself, or Tozier, or even Chet for this. Having noticed this problem, perhaps a bit later than I might wish, I’m going to clean it up a bit. And if it happens to you, I’m not going to blame you, or even the Dark Scrum™ manager who put you under pressure. I’m just going to suggest that you try to be sensitive to when things are getting sticky, and clean things up.

Anyway, those are my thoughts for the day. We’ll be back to this next Tuesday. End to end, I’m almost sure :)

  1. “Brighton Agile Roundtable”, our name for the round table we use when visiting the Barnes&Noble coffee shop in Brighton. Michigan, that is. ↩