Tuesday

For personal reasons, it has been quite a while since we wrote any code. Let’s see if we can do some today.

We think the grand picture today is to work toward having a manifest of “new” folders and files, so that we can build the system and then publish the new material on my site.

We review the tests:

require "minitest/autorun"
require "./jekyllrunner"

# to make this work on my iMac (and I swear it used to), I had to re?enable FTP
# first in System Preferences / Sharing, allow Remote Login (which I am sure used to be on) and then
# in Terminal: sudo -s launchctl load -w /System/Library/LaunchDaemons/ftp.plist
# Which allegedly turns on the FTP, per 
# https://gaborhargitai.hu/enable-built-in-ftp-server-in-mac-os-x-yosemite-el-capitan-sierra/

# all this does seem a bit wide open but I suppose my iMac doesn't have a public address

# Then I had to move the project around and re-clone it. Now seems to work.

class Test_JekyllRunner  < Minitest::Test

  def setup
    @test_startup_folder = Dir.pwd
  end

  def teardown
    Dir.chdir(@test_startup_folder)
  end

  # check_contents:
    # √ the YAML worked
    # √ the jpegs are still viewable
    # √ categories are done and uploaded
    # feed.xml is uploaded
    # index.html is uploaded

  def set_up_for_jekyll_testing
    FileUtils.rm_rf("./_target")
    FileUtils.mkdir("./_target")
    FileUtils.mkdir("./_target/articles")
    FileUtils.mkdir("./_target/categories")
    FileUtils.rm_rf("./test_jekyll_site/articles")
    ipad_folder = '/Users/ron/Dropbox/_source'
    jekyll_folder = './test_jekyll_site'
    ftp_host = 'localhost'
    ftp_target_folder = 'programming/test-ftp/_target/'
    password_prefix = 'TEST_'
    return JekyllRunner.new(ipad_folder, jekyll_folder, 
      ftp_host, ftp_target_folder, password_prefix)
  end

  def set_up_for_really_jekylling
    # root here is /Users/ron/programming/test-ftp
    new_files_folder = '/Users/ron/Dropbox/_articles_from_ipad'
    jekyll_folder = '/Users/ron/programming/rj-com'
    ftp_host = 'ftp.ronjeffries.com'
    ftp_target_folder = 'httpdocs/'
    password_prefix = 'PROD_'
    return JekyllRunner.new(new_files_folder, jekyll_folder, 
      ftp_host, ftp_target_folder, password_prefix)
  end

  # def test_in_live
  #   jr = set_up_for_really_jekylling
  #   jr.run
  # end

  # following test we checked manually and now trust it
  # cf. hold my beer

  # def test_move_test_file_to_site
  #   jr = set_up_for_really_jekylling
  #   testfile = "#{jr.jekyll_folder}/_site/test-file-do-not-read.txt"
  #   FileUtils.touch(testfile)
  #   jr.move_test_file
  # end
  
  def test_file_copy
    jr = set_up_for_jekyll_testing
    assert(!File.exist?(
      './test_jekyll_site/articles/a.txt'), 
      "mistakenly found a.txt")
    jr.move_ipad_files
    assert(File.exist?(
      './test_jekyll_site/articles/a.txt'), 
      "can't find a.txt")
  end

  def test_jekyll_run
    jr = set_up_for_jekyll_testing
    jr.move_ipad_files
    jr.run_jekyll
    assert(File.exist?(
      './test_jekyll_site/_site/articles/a.txt'), 
      "can't find jekyllated a.txt")
    assert(File.exist?(
      './test_jekyll_site/_site/articles/subfolder/index.html'),
      "can't find jekyllated subfolder")
  end

  # def test_we_know_what_to_ftp
  #   jr = set_up_for_jekyll_testing
  #   jr.move_ipad_files
  #   expected_to_ftp = ["a.txt", "anotherfolder", 
  #     "anotherfolder/f5.txt", "anotherfolder/index.html", 
  #     "b.txt", "pic.JPG", "subfolder", "subfolder/index.html", 
  #     "subfolder/sf1.txt", "subfolder/st2.txt"]
  #   assert_equal(expected_to_ftp, jr.files_to_ftp('something goes here'))
  # end

  def test_end_to_end
    jr = set_up_for_jekyll_testing
    jr.run
    assert(File.exist?('./_target/articles/anotherfolder/f5.txt'), 
      "end to end can't find ftped f5.txt from #{Dir.pwd}")  
  end

  def test_categories_are_uploaded
    jr = set_up_for_jekyll_testing
    jr.run
    sought = './_target/categories/cat-1/index.html'
    assert(File.exist?(sought), 
      "end to end can't find ftped #{sought}")  
  end

  def test_feed_xml_is_uploaded
    jr = set_up_for_jekyll_testing
    jr.run
    sought = './_target/feed.xml'
    assert(File.exist?(sought), 
      "end to end can't find ftped #{sought}")  
  end

  def test_index_html_is_uploaded
    jr = set_up_for_jekyll_testing
    jr.run
    sought = './_target/index.html'
    assert(File.exist?(sought), 
      "end to end can't find ftped #{sought}")  
  end

  def test_setup_teardown_restores_chdir
    result = (`pwd`).chomp
    assert_equal(@test_startup_folder, result)    
  end
    
  def test_chdir_affects_where_backtick_runs
    # note we need not save and restore pwd because setup/teardown do
    Dir.chdir('/Users/ron')
    result = (`pwd`).chomp
    assert_equal('/Users/ron', result)
  end
end

I considered removing all the odd comments from the file, but it seems to me that often, in real code, we find strange commentary, trying to explain things or provide context of some kind. Our hope, I suppose, is that the next time we’re confused, we’ll find a comment and it will help us. As I read these now, I merely find them distracting. In the spirit of The Book of Erors, I’ve decided to leave them here. Think about whether they help you, or get in your way, as you try to follow along.

Moving right along, we observe from our tests that we are now unconditionally moving everything from the iPad input folder into the Jekyll source folder. We foresee that our date-driven manifest will limit the files we actually publish. So far so good.

– we believe – we’re using the list of files moved from iPad as our manifest. We’ll check to be sure we believe correctly, and of course that manifest is what we need to change.

We discover this:

  def move_the_articles
    move_batch("#{@jekyll_folder}/_site/articles", 
      'articles', files_to_ftp(@ipad) )
  end

  def files_to_ftp(folder)
    perform_in_folder(folder) do
      Dir.glob('**/*').collect { |f| rename_md_to_html(f) }
    end
  end

We extracted files_to_ftp because it’s used to move the category index files and probably other items. However … is it in fact move_the_articles that needs to be changed? If that method moved the new articles, wouldn’t it be just what we want?

So I propose that we have a simple “fix” here. We’ll just rename this function!

  def move_the_new_articles
    move_batch("#{@jekyll_folder}/_site/articles", 
      'articles', files_to_ftp(@ipad) )
  end

Super! The tests of course all run … for now. So now, what to do to make it work. Presumably … we need a test.1

What, Tozier asks, is the functionality we want here? Is it that we would like to move all the files that are newer than a certain time? (Yes, it is!) The task is to find the things that are newer in the source folder and then to move the corresponding rendered files in the Jekyll output folder.

We find ourselves nearly understanding how to do this. We want to just type it in. Oddly enough, I am the one disciplined enough to demand that we write a test … this time.

We begin by renaming the test_file_copy test, resulting in

  def test_ipad_file_copy
    jr = set_up_for_jekyll_testing
    assert(!File.exist?(
      './test_jekyll_site/articles/a.txt'), 
      "mistakenly found a.txt")
    jr.move_ipad_files
    assert(File.exist?(
      './test_jekyll_site/articles/a.txt'), 
      "can't find a.txt")
  end

We sketch:

  def test_find_new_files
    # set some as-of date
    # set some new articles
    # check whether our manifest-maker returns them
  end

We review our test setup, which creates some files.

We also need to set the as-of date, which we intend to have be the date-time when Jekyll last built the site. For now, Tozier suggests that it’s simply an argument to whatever function we write to do the job.{:.aside}

We check our current setup:

  def set_up_for_jekyll_testing
    FileUtils.rm_rf("./_target")
    FileUtils.mkdir("./_target")
    FileUtils.mkdir("./_target/articles")
    FileUtils.mkdir("./_target/categories")
    FileUtils.rm_rf("./test_jekyll_site/articles")
    ipad_folder = '/Users/ron/Dropbox/_source'
    jekyll_folder = './test_jekyll_site'
    ftp_host = 'localhost'
    ftp_target_folder = 'programming/test-ftp/_target/'
    password_prefix = 'TEST_'
    return JekyllRunner.new(ipad_folder, jekyll_folder, 
      ftp_host, ftp_target_folder, password_prefix)
  end

This setup presently completely clears the (test) Jekyll source folders, and the tests presently anticipate that only the files from the iPad folder will be there. T suggests creating some new articles into the Jekyll source in our test, and working from there. Makes sense to me. We’ll have to fiddle the dates somehow.

Let’s expand our test sketch:

  def test_find_new_files
    # set some as-of date
    # create an article folder and files older than as-of
    # create an article folder and files newer than as-of
    # check whether our manifest-maker returns the newer only
  end

We expand further:

  def test_find_new_files
    # set some as-of date
    as_of = DateTime.new(2017,10,1,6,0,0)
    # create an article folder and files older than as-of
    assert(!File.exist?("./test_jekyll_site/articles/older"))
    older_path = Pathname.new("./test_jekyll_site/articles/older")
    older_path.mkdir
    assert(File.exist?("./test_jekyll_site/articles/older"))
    # create an article folder and files newer than as-of
    # check whether our manifest-maker returns the newer only
  end

Now we believe, though we’ve not seen it happen, that we have created a folder called older. Kind of an older folder if you will. As it were. But I digress.

But it’s not older, which we’ll have to do something about. First, though, a file.

We get so far as this before Tozier recognizes a problem:

  def test_find_new_files
    # set some as-of date
    as_of = DateTime.new(2017,10,1,6,0,0)
    # create an article folder and files older than as-of
    assert(!File.exist?("./test_jekyll_site/articles/older"))
    older_path = Pathname.new("./test_jekyll_site/articles/older")
    older_path.mkdir
    assert(File.exist?("./test_jekyll_site/articles/older"))
    older_file = older_path + "index.md"
    older_file.write("Hello, World!")
    exit
    # create an article folder and files newer than as-of
    # check whether our manifest-maker returns the newer only
  end

We think we’re writing this file to Jekyll’s output folder. At least part of our problem is the poor names for these folders. It seems that we could rename these if we are careful. And these repeated names should be saved in some central single place anyway.

We set out to try this but realize that we’re not writing to the output folder after all. Our test Jekyll input is test_jekyll_site, and the output is test_jekyll_site/_site. This is very confusing. Note also that I’ve changed the real site generation to put the output outside of the Jekyll source, so we’ll encounter this issue again.

OK, reset. We don’t need to change our test. (I want to confess right here that we did halt the tests and look at the folders to be sure our little file-writing exercise worked. Let’s move on with the manifest test and set the dates and the new files.)

After some confusion we reach:

  def test_find_new_files
    # set some as-of date
    before = DateTime.new(2017,10,1,5,0,0)
    as_of = DateTime.new(2017,10,1,6,0,0)
    after = DateTime.new(2017,10,1,7,0,0)
    # create an article folder and files older than as-of
    assert(!File.exist?("./test_jekyll_site/articles/older"))
    older_path = Pathname.new("./test_jekyll_site/articles/older")
    older_path.mkdir
    assert(File.exist?("./test_jekyll_site/articles/older"))
    older_file = older_path + "index.md"
    older_file.write("Hello, World!")
    FileUtils.touch older_file, :mtime => before.to_time
    assert(File.exist?("./test_jekyll_site/articles/older/index.md"))
    assert(older_file.mtime < as_of.to_time, "file #{older_file.mtime} < #{as_of.to_time} should be older")
    # create an article folder and files newer than as-of
    # check whether our manifest-maker returns the newer only
  end

This manages to create an article folder, older and an article within it, and to set the modification time of the article to older than our as_of time. So far so good.

We are running short of time. We note that our current test could better be named:

  def test_create_old_and_new_files
    # set some as-of date
    before = DateTime.new(2017,10,1,5,0,0)
    as_of = DateTime.new(2017,10,1,6,0,0)
    after = DateTime.new(2017,10,1,7,0,0)
    # create an article folder and files older than as-of
    assert(!File.exist?("./test_jekyll_site/articles/older"))
    older_path = Pathname.new("./test_jekyll_site/articles/older")
    older_path.mkdir
    assert(File.exist?("./test_jekyll_site/articles/older"))
    older_file = older_path + "index.md"
    older_file.write("Hello, World!")
    FileUtils.touch older_file, :mtime => before.to_time
    assert(File.exist?("./test_jekyll_site/articles/older/index.md"))
    assert(older_file.mtime < as_of.to_time, "file #{older_file.mtime} < #{as_of.to_time} should be older")
    # create an article folder and files newer than as-of
    # check whether our manifest-maker returns the newer only
  end

We can extend it to create the newer file and then we’ll take a break until tomorrow or Thursday. Here’s our final test for the day:

  def test_create_old_and_new_files
    # set some as-of date
    before = DateTime.new(2017,10,1,5,0,0)
    as_of = DateTime.new(2017,10,1,6,0,0)
    after = DateTime.new(2017,10,1,7,0,0)
    # create an article folder and files older than as-of
    assert(!File.exist?("./test_jekyll_site/articles/older"))
    older_path = Pathname.new("./test_jekyll_site/articles/older")
    older_path.mkdir
    assert(File.exist?("./test_jekyll_site/articles/older"))
    older_file = older_path + "index.md"
    older_file.write("Hello, World!")
    FileUtils.touch older_file, :mtime => before.to_time
    assert(File.exist?("./test_jekyll_site/articles/older/index.md"))
    assert(older_file.mtime < as_of.to_time, "file #{older_file.mtime} < #{as_of.to_time} should be older")
    # create an article folder and files newer than as-of
    newer_path = Pathname.new("./test_jekyll_site/articles/newer")
    newer_file = newer_path + "index.md"
    newer_path.mkdir
    newer_file.write("Hello, New World!")
    FileUtils.touch newer_file, :mtime => after.to_time
    assert(newer_file.mtime > as_of.to_time, "should be newer")
    # check whether our manifest-maker returns the newer only
  end

Looking back at the day, we’re impressed by how the Pathname approach cleans up the code for us. There are glitches, like the fact that you can’t compare a DateTime to a Time, and such, but this code is less messy and less fragile than the earlier code we wrote using File and Dir. “So that’s nice.”

This feels good for going forward and Tozier points out that using mtime in our actual code will turn out to be the right thing, not ctime.

As sessions go, we’ve spent less time confused than usual. This test was a bit more micro than others we’ve done, which helped. Certainly we have no reason to assume that we’re any smarter than a few weeks ago.

I feel this code is still a bit procedural, with maybe some domain objects trying to come out. And we need to refactor out these string names.

See you … Wednesday

Wednesday - Time Flies!

Beginning anew on Wednesday, we discuss the fact that our current “manifest” has a list of folders and files within that folder. It seems to me (and perhaps us) that we could have a manifest that was a list of “folders containing at least one new file”, and we could have the publication code decide which files to move from within that folder, or just move them all, which wouldn’t be much more costly.

Tozier suggests that we “look at such a list”, by which he means … what happens when a glob is filtered by date > some date. (Aside: I think he means he wants to fiddle with this idea and see what happens. I’m good with that.)

We don’t know how to start with a test here. We’ll try some IRB …

We quickly come up with this:

files = Pathname.glob("**/*")
cutoff = Time.new(2017,10,1)
keepers = files.select { |e| e.file? && e.mtime > cutoff }

Which delivers this:

[#<Pathname:articles/017-08ff/hiring/index.md>, #<Pathname:articles/017-08ff/ipad-n/index.md>]

This, conveniently, is the list of Pathnames representing all files inside the Jekyll source which are newer than the cutoff date. It seems clear that even we, ignorant as we are, could write code that will use this information to create the necessary folders and files on the published site.

Do we need to write a new test, or can we put in this new implementation and make our existing tests tell us that we’re OK? Our present FTP code looks like this:

  def move_batch(local_site_folder, ftp_target_folder, folders_and_files )
    perform_in_folder(local_site_folder) do 
      ftp_connection = ftp_connected_to_target
      ftp_connection.passive = true
      ftp_connection.chdir(ftp_target_folder)
      folders_and_files.each do |file|
        ftp_one_file(ftp_connection, file)
      end
      ftp_connection.close
    end
  end

  def ftp_one_file(ftp, file_name)
    if File::directory? file_name
      ftp_mkdir_safely(ftp, file_name)
    else
      ftp.putbinaryfile(file_name,file_name)
    end
  end

What we have here actually looks to see whether the thing being published is a folder name or a file name and creates the folder if need be. In our new scheme, we plan to have only the file names, so we will have to make the folder by pulling the path part out of the Pathname object.

Interruption – Suddenly it’s another day

We experienced an interruption right in the middle of this activity. We’re simulating a real project just a bit more faithfully than I might like. Anyway, here we are at our next session, and Tozier will be here soon.

I was thinking this morning about how this little product has evolved over the not very many hours (but many days) we’ve worked on it. I am hatching a new evolution, and part of me thinks we should just get something done, while part of me thinks we should do the closest to the right thing we can manage. I’m aware, however, that my nature seems to always be to want the new thing, and that this can sometimes get in the way of getting things done. So I try to resist the urge to change for the sake of change. We’ll discuss that when T gets here.

My thoughts this morning led me to recall Stewart Brand’s book, How Buildings Learn. Published back in 1994, the book describes how buildings evolve over time, to meet changing needs. I recall that many of us were enamored of the book at the time, because it reminded us of refactoring and (I suppose) reusing code. Today, the lesson I’m drawing is that the product we build here should be the one most suitable to our need, not necessarily the one we first envisioned.

The Idea

In a separate effort, I’ve already moved my Jekyll source folder to Dropbox. I did this because Dropbox synchronizes my home computer and my laptop quite nicely, so that I don’t have to go through the git commit / git pull cycle when I switch computers. (I’ll still push things to GitHub as an archive.)

With the Jekyll source in Dropbox, there’s no need for the separate iPad staging folders that we copy down into Jekyll source before running Jekyll. Instead, the rule can be that when I’m editing in iPad, I edit the real article in the iPad’s image of the source, and when Dropbox syncs to whatever computer is running this app, the site will be built and published, voila!

Tozier, surprisingly enough, agrees with this idea. It will simplify our program here, which can’t hurt, and (I believe) we’ll have no extra work to do to make it happen. (Some of the work might be changed in detail, such as which files to look at for trigger dates and the like, but that’s not going to be more work, just different work.)

“However”, says Tozier, with his finger in the air … “we have to remember that Jekyll’s output folder is not in Dropbox, and we’ll have to be sure to look in the right place to find the files to be published.” Agreed.

Now then … should we proceed by refactoring and reworking the existing project, or should we create a clean slate, and build on what we’ve learned, “judiciously” adopting any tests and code we find useful? The latter seems tastier, but I have learned over the years that this thinking is my nemesis.

What will our intrepid heroes do? Tune in next time for the thrilling next episode.

  1. If you’re wondering what all the ellipses are … those are places where I’m pausing to think as I write down what Tozier and I are doing. ↩