It has been a week since we got together and therefore a week since we had any idea what we were doing. We see from the last article that we were trying to build some kind of a mapping from input names to output names, for the FTP code to use. We remain confused. Now we’ll read and run the tests and see what they tell us.

Here’s the code and tests we have. I’d suggest that you just skim them for now.

require "minitest/autorun"
require "net/ftp"
require "./passwords"
require "set"

class Test_JekyllRunner  < Minitest::Test

  def setup
    @test_startup_folder = Dir.pwd
  end

  def teardown
    Dir.chdir(@test_startup_folder)
  end

  # copy _source to working_folder/articles
  # run jekyll (in working_folder)
  # ftp only the copied and jekylled files from working_folder/_site/articles to remote site
  # with <any>.md -> <any>.html

  # check_contents:
    # the YAML worked
    # the jpegs are still viewable
    # categories are done and uploaded

  def set_up_for_jekyll_testing
    FileUtils.rm_rf("./test_jekyll_site/articles")
    ipad_folder = '/Users/ron/Dropbox/_source'
    jekyll_folder = './test_jekyll_site'
    ftp = nil 
    return JekyllRunner.new(ipad_folder, jekyll_folder, ftp)
  end
  

  def test_file_copy
    jr = set_up_for_jekyll_testing
    assert(!File.exist?('./test_jekyll_site/articles/a.txt'), "mistakenly found a.txt")
    jr.move_ipad_files # sorry, Demeter
    # this file down here is in jekyll input
    assert(File.exist?('./test_jekyll_site/articles/a.txt'), "can't find a.txt")
  end

  def test_jekyll_run
    jr = set_up_for_jekyll_testing
    jr.move_ipad_files # sorry, Demeter
    jr.run_jekyll # this is the big deal right here
    # these files down here is in jekyll output
    assert(File.exist?('./test_jekyll_site/_site/articles/a.txt'), "can't find jekyllated a.txt")
    assert(File.exist?('./test_jekyll_site/_site/articles/subfolder/index.html'), "can't find jekyllated subfolder")
  end

  def test_we_know_what_to_ftp
    jr = set_up_for_jekyll_testing
    jr.move_ipad_files # sorry, Demeter
    # jr.run_jekyll 
    expected_to_ftp = ["a.txt", "anotherfolder", 
      "anotherfolder/f5.txt", "anotherfolder/index.html", 
      "b.txt", "pic.JPG", "subfolder", "subfolder/index.html", 
      "subfolder/sf1.txt", "subfolder/st2.txt"]
    assert_equal(expected_to_ftp, jr.what_to_ftp)
  end

  # def test_what_will_be_ftped
  #   FileUtils.rm_rf("./test_jekyll_site/articles")
  #   ipad_folder = '/Users/ron/Dropbox/_source'
  #   jekyll_folder = './test_jekyll_site'
  #   ftp = nil 
  #   jr = JekyllRunner.new(ipad_folder, jekyll_folder, ftp)
  #   jr.move_ipad_files # sorry, Demeter
  #   moved = jr.ftp_the_files(simulated = true) 
  #   expected_to_move = []
  #   assert_equal(expected_to_move, moved)
  # end

  # def test_end_to_end
  #   FileUtils.rm_rf("./test_jekyll_site/articles")
  #   ipad_folder = '/Users/ron/Dropbox/_source' 
  #   jekyll_folder = './test_jekyll_site'
  #   ftp = nil 
  #   jr = JekyllRunner.new(ipad_folder, jekyll_folder, ftp)
  #   jr.run
  #   assert(File.exist?('./_target/articles/anotherfolder/'), "can't find jekyllated a.txt")    
  # end

  def test_setup_teardown_restores_chdir
    result = (`pwd`).chomp
    assert_equal(@test_startup_folder, result)    
  end
    
  def test_chdir_affects_where_backtick_runs
    Dir.chdir('/Users/ron')
    result = (`pwd`).chomp
    assert_equal('/Users/ron', result)
  end
end

class JekyllRunner
  def initialize(ipad_folder, jekyll_folder, ftp_to_site)
    @ipad = ipad_folder
    @jekyll_folder = jekyll_folder
    @ftp = ftp_to_site
  end

  def ftp_connected_to_target
    ftp = Net::FTP.new('localhost')
    ftp.login(Passwords::USER,Passwords::PASSWORD)
    ftp.chdir('programming/test-ftp/_target/articles')
    ftp
  end

  def move_ipad_files
    FileUtils.cp_r("#{@ipad}/.","#{@jekyll_folder}/articles")
  end

  def what_to_ftp
    pwd = Dir.pwd
    Dir.chdir(@ipad)
    files = Dir.glob('**/*')
    Dir.chdir(pwd)
    files.collect { |f| rename(f) }
  end

  def ftp_the_files(simulated = false)
    pwd = Dir.pwd
    Dir.chdir("#{@jekyll_folder}/_site/articles")
    # position Ftp
    ftp_connection = ftp_connected_to_target
    result = []
    files = what_to_ftp
    files.each do |file|
      result << ftp_one_file(ftp_connection, file, simulated)
    end
    result
  end

  def ftp_one_file(ftp, file_name, simulated)
    unless simulated
      # if File::directory? file_name
      #   ftp_mkdir_safely(ftp, file_name)
      # else
      #   ftp.putbinaryfile(file_name,file_name)
      # end
    end
    return "#{Dir.pwd}/#{file_name} to #{ftp.pwd}/#{file_name}"
  end

  def rename(file_name)
    file_name.gsub(/\.md$/,".html")
  end

  def run_jekyll
    pwd = Dir.pwd # TODO fix this to be more reasonable
    Dir.chdir(@jekyll_folder)
    `jekyll build`
    Dir.chdir(pwd)
  end

  def run
    move_ipad_files
    run_jekyll
    # ftp_the_results_to_site
  end
end

There are a number of commented-out tests here, but we think they are decent sketches of what has to happen, or tests we may need to write.

Our minds are refreshed: We think we have code that creates approximately the right list of files to FTP to the site. Let’s work toward testing whether they move correctly.

First we do some renaming and uncommenting, producing this test:

  def test_what_will_be_ftped
    jr = set_up_for_jekyll_testing
    jr.move_ipad_files
    moved = jr.ftp_the_files(simulated = true) 
    expected_to_be_ftped = []
    assert_equal(expected_to_be_ftped, moved)
  end

We’ve left the expected value blank, which is our practice when we produce a collection. We eyeball it and then put that value into the test. This is arguably not a good practice, since it’s error-prone, but right now we’re not smart enough to do better.

Making this run required some hackery in JekyllRunner to keep it from accessing an FTP object while simulating:

  def what_to_ftp
    pwd = Dir.pwd
    Dir.chdir(@ipad)
    files = Dir.glob('**/*')
    Dir.chdir(pwd)
    files.collect { |f| rename(f) }
  end

  def ftp_the_files(simulated = false)
    pwd = Dir.pwd
    Dir.chdir("#{@jekyll_folder}/_site/articles")
    # position Ftp
    ftp_connection = ftp_connected_to_target unless simulated
    result = []
    files = what_to_ftp
    files.each do |file|
      result << ftp_one_file(ftp_connection, file, simulated)
    end
    result
  end

  def ftp_one_file(ftp, file_name, simulated)
    unless simulated
      # if File::directory? file_name
      #   ftp_mkdir_safely(ftp, file_name)
      # else
      #   ftp.putbinaryfile(file_name,file_name)
      # end
    end
    pwd = simulated ? "something" : ftp.pwd
    return "#{Dir.pwd}/#{file_name} to #{pwd}/#{file_name}"
  end

Aside: As I clean up this article from notes, I notice that the what_to_ftp function saves the working folder, changes it, and then doesn’t restore it. I remember that this causes us trouble further down the line. We didn’t notice it at the time we wrote it.

What is arguably ironic is that Tozier and I have spoken several times about the need to do something about the repetition of the save and (forget to) restore folder pattern.

We manage to make that test run, and quickly replace it:

  def test_what_will_be_ftped
    jr = set_up_for_jekyll_testing
    jr.move_ipad_files
    list_of_allegedly_moved_things = jr.ftp_the_files(simulated = true) 
    assert list_of_allegedly_moved_things.include? "/Users/ron/programming/test-ftp/test_jekyll_site/_site/articles/subfolder/index.html to something/subfolder/index.html"
  end

This makes a bit more sense than checking a huge list for equality. We get a list of lots of things, and just check one item to be sure it’s in there. The information returned is ugly but it’s what we wanted to see: where are you really moving from, where are you really going to move it to? If we wanted more confidence, we’d check a few more items explicitly, but we’re happy for now.

We’re getting closer and closer to the inevitable moment when we try to move some files, but we’re also getting code that I like less and less. We really shouldn’t have to do all this checking of simulated but I’m not really interested in making a fake FTP object. Maybe, when we finally do the end to end test that’s coming up, we can pass in a real FTP object, our local one, and take out some of the simulation flags. They were just put in there to let us test incrementally.

We pause to chat …

Is this the mangle which we see before us? Does it even have a handle?
– Wm. Shakespeare, probably

We’re really feeling more and more nervous about all this. It seems grubby and weird and we’re not sure why. We expected to feel pretty good by now. What’s going on?

Tozier recalls having premonitions about using this mysterious FTP library thing. Is our experience now related to that? We haven’t even touched FTP yet! But are we leaning away from it in some way, and thus off balance? We’ve certainly been tiptoeing around FTP with these simulation flags and checking lists of files.

Tozier also reminds us that as Detroit School TDDers, we avoid using mocks and fakes as much as we can. Here, we’re kind of faking or mocking. Actually, I think we’re faking, and the code is mocking us.

One fear here is that I’m scared of writing a test that might destroy my actual web site. This fear isn’t particularly valid, because we know how to write an FTP that runs internally to my laptop. Still, this fear is probably helping us mess up the code and tests.

Be all that as it may, this seemingly simple application seems to be pushing back. We can press on, or try to clean it up now. I lean toward pressing on, for two reasons. One is that I think that once we get an end to end test, we can remove some of these intermediates. The other, not so worthy, is that I often feel like pressing on when anyone sensible would back off and do something more sensible.

I’m gonna press on. I freely grant that often, when I do this, it’s a mistake. I’m sure everything will be OK this time.

Tozier notes two things that our next test should do: check that the files we want moved actually move, and check that things that already exist over there but were not to be moved don’t get destroyed.

I’m putting this note here so that we can revisit these after we do whatever we do, because I’m expecting not to worry about the same things. This is one of those cases where one of us is confident and the other is not. I was taught that in those cases one should write the test. I find myself arguing against some of the best advice I’ve ever had. How odd …

Then press on …

We’re going to need to create a real FTP object, connected to a safe folder on my Mac. We have this code from our separate learning tests:

  def ftp_connected_to_target
    ftp = Net::FTP.new('localhost')
    ftp.login(Passwords::USER,Passwords::PASSWORD)
    ftp.chdir('programming/test-ftp/_target')
    ftp
  end

This is close to what we need. We could call a similar function from our existing setup, where now we set ftp to nil. (We’ll have to deal with the simulation flag, probably.)

  def set_up_for_jekyll_testing
    FileUtils.rm_rf("./test_jekyll_site/articles")
    ipad_folder = '/Users/ron/Dropbox/_source'
    jekyll_folder = './test_jekyll_site'
    ftp = nil 
    return JekyllRunner.new(ipad_folder, jekyll_folder, ftp)
  end

I’ll move the code over and we’ll look at it. We wind up with this:

  def set_up_for_jekyll_testing
    FileUtils.rm_rf("./_target")
    FileUtils.mkdir("./_target")
    FileUtils.mkdir("./_target/articles")
    FileUtils.rm_rf("./test_jekyll_site/articles")
    ipad_folder = '/Users/ron/Dropbox/_source'
    jekyll_folder = './test_jekyll_site'
    ftp = ftp_connected_to_target 
    return JekyllRunner.new(ipad_folder, jekyll_folder, ftp)
  end

  def ftp_connected_to_target
    ftp = Net::FTP.new('localhost')
    ftp.login(Passwords::USER,Passwords::PASSWORD)
    ftp.chdir('programming/test-ftp/_target')
    ftp
  end

WARNING: There are lies above!

The above simple tests took nigh on to an hour to make run. This isn’t one of those “and then a miracle occurs and we suddenly show a fairly good looking test and it actually runs” situations. This took an hour!!

One big issue was that our test was looking into the wrong folder to see whether a.txt was there. Because the tests create and tear down the target folders, we couldn’t just look to see if they were OK, so we spent a long time trying to print things or see things without tearing apart the tests.

Once we found that, a smaller issue was that a.txt wasn’t supposed to be there anyway, but f5.txt was. Copy-paste error anyone? Known not to do that for decades, anyone? Grrr.

But the underlying issue is that both the working directory in the file system, and the working directory in our FTP object tend to jump around. This is supposedly for our convenience, or at least to accommodate the weird way the file copying and testing commands work, but the result is that we never know where we are and it seems like we’re never where we want to be.

An artifiical voice says “Recalculating…”

We could, and perhaps should, set up some constant strings to use in setting things to point to the right places. We’re still seeing evidence that the folder navigation is troubling us, and clearly our tiny little brains can’t cope.

However, our tests are actually running. We have successfully moved files from the source Dropbox folder, down into our fake Jekyll site, generated the site output, and moved the generated output to our fake web site. It’s working, end to end!

Do you hear me??? It’s working!! It’s ALIVE!!

We could strip out a lot of our temporary tests, relying on our now working end to end test, then clean up what’s left, or just back away slowly, as one does. We choose to back away slowly, and to retain these intermediate tests at least until a final refactoring, if not forever. Probably we should leave most of them forever, as they reflect and test the architecture.

Right now we are sure that the current code won’t work with the real site, because none of the folders involved are parameters to the constructor of JekyllServer. (And the credentials file isn’t ready either.)

This could be left to the next round of testing somehow. At the moment, though, I’m not quite sure what kind of test I could write that tests whether it will work if pointed at the real site, without pointing it at the real site and praying. Some of the existing tests, and the simulated flag, are intended to tell us whether it probably will work …

Wait! Tozier insists on setting our test that uses the long-form path back to using the sensible local path, and fixing whoever is hammering the working directory. We find that and fix it:

  def test_end_to_end
    jr = set_up_for_jekyll_testing
    jr.run
    assert(File.exist?('./_target/articles/anotherfolder/f5.txt'), 
      "end to end can't find ftped f5.txt from #{Dir.pwd}")  
  end

We had made that test run by putting the long-form path,, /users/ron/blah/mumble in there. It worked but Tozier was right to want to change it back. The fix, of course, was to find and fix the missing resetting of the Dir folder, in one or both of these bits:

  def what_to_ftp
    pwd = Dir.pwd
    Dir.chdir(@ipad)
    files = Dir.glob('**/*')
    Dir.chdir(pwd)
    files.collect { |f| rename(f) }
  end

  def ftp_the_files(simulated = false)
    pwd = Dir.pwd
    Dir.chdir("#{@jekyll_folder}/_site/articles")
    # position Ftp
    ftp_connection = ftp_connected_to_target unless simulated
    result = []
    files = what_to_ftp
    files.each do |file|
      result << ftp_one_file(ftp_connection, file, simulated)
    end
    Dir.chdir(pwd)
    result
  end

All the tests that should run are now running. We have an end-to-end test that copies files down from an input folder into a Jekyll folder, runs Jekyll, and pushes the corresponding _site files (and only those) to a designated FTP site. We’re nearly there, for values of nearly. Remaining to do is to move the appropriate category indexes and the main index file. (And, I suppose whatever we don’t even remember.)

More immediately, we have identified a design problem. What we need is an end-around function that saves the existing pwd, executes some block, and restores the old pwd. We can’t think of the name of that trick but we know how to do it. This, too, we’ll leave for next time.

Reflection on Erors™

This series of articles is part of my “Book of Erors™” series. Whenever I show the creation of code, I always show the dirty bits, because I believe it’s always dirtier than it looks, and that we all need to remember that. New folks need to realize that their horrible lost feelings are normal, and older folks need to become as sensitive as possible to those feelings, because they’re telling us something.

Of course, as I’ve said before, maybe I’m the only one who gets into these situations, in which case feel free to feel really good about yourself in comparison. That’s fine, too.

Now this is really a nearly trivial program. Some people argued that it could be done with a bash script or with their favorite build system. They are probably right. My mission is to write about programming, so I chose to program it. My work style on this, and most things, is to work for a few hours every few days. It’s probable that any of my readers could have just whipped this out in a day or less. (If you agree, try it, from scratch, and let me know.) But working only two days a week or less, for two or three hours at a time, I get to forget things that I’d be carrying in my head over the course of a day. And that means I get to rediscover them in a few days. It’s just like real programming, with legacy code that we don’t quite remember any more.

I’m suggesting that what goes on here is very much like “real” programming. It certainly feels similar to me. And I learn the same things, over and over again, things like these:

  • If I don’t know quite how something works, a saved test is better than a quick experiment and turning to the real code to paste it in.
  • If something seems too easy to need a test, my hackles should go up. I might go ahead, but stay alert for the error that a test would have prevented.
  • If something seems too hard to test, it’s almost always too hard to write without a test.

I’ve been programming longer than most of you have been alive. Whenever I’m faced with any programming problem, it seems that a sensible modular approach to it comes right to mind. The sensible modular approach we have here, of course is:

  1. Put iPad articles in a Dropbox folder;
  2. From the Dropbox folder, copy to the Jekyll folder, remembering what you did;
  3. Run Jekyll, creating the _site folder;
  4. FTP the _site files corresponding to the input files up to my web site;
  5. FTP all the category files up, and the main index.html.

Well not quite. In truth I immediately foresaw the rough shape of this but an understanding of the extra files, for categories and index, came later and in a fuzzy form. Nonetheless, I was never at a loss as to how to approach this problem, in a modular-enough way that everything would fit in.

It’s almost always like that. I just know what has to be done.

And yet … it never, ever goes just like that. There are always things that get in the way. Things I hadn’t thought of, should have thought of, couldn’t even have thought of, or just things that happen. Once in a while there’s even a good surprise, like how easy it was to set up a Jekyll build to test with1.

My mission, in these Erors™ articles, is to keep fresh in my mind, and in yours, if you need it, how many balls we’re keeping in the air when we program, how many balls are flying at us from behind, and how many balls we completely forgot to bring with us to the circus. The big learning for me from all this “Agile” stuff, is that working in very small steps, from working software to not quite working and back to working, is best for me. It means that I don’t so often go down a path of destruction for days or weeks at a time2. And when that software is well-supported with tests, I go faster, even though the tests are often tedius to write.

For me, this is the best way I know to work. I commend the idea to you as something to try, so that you can better find your favorite place to stand amid all this software. I expect you’ll find similar results to mine, but even if you don’t, working with these ideas may help you find your own place.

See you next time!

  1. These missed ideas, by the way, are part of why, in my recent Science article, I’m quite pessimistic about ever really getting a suitable estimate for even the simplest of projects. ↩

  2. I do still go down ratholes for an hour at a time, and even that’s too long. Maybe I need a twenty-minute timer? ↩