Let’s try making our learning tests more clear, and maybe drive out some methods for an FTP-like object. Here’s the code as it stands this morning at BAR. Hasn’t changed since I wrote it at 4 AM. That’s nice.

require "minitest/autorun"
require "net/ftp"
require 'pathname'

# The code here is pretty close to the simplest thing that will work;
# Or, in the case of some tests ... not work.

class Test_FTP  < Minitest::Test
  def test_start
    path = 'Dropbox/ftp-experiments'
    ftp = Net::FTP.new('localhost')
    ftp.login(user="ron", passwd="PASSWORD")
    assert_equal('/Users/ron', ftp.pwd)
    ftp.chdir(path)
    top = ftp.pwd
    assert_equal('/Users/ron/Dropbox/ftp-experiments', ftp.pwd)
    assert(folder_exists?(ftp, 'first'))
    assert(! folder_exists?(ftp,'nowhere'))
    begin
      ftp.mkdir('first')
    rescue Net::FTPPermError => e
      assert(e.message.match('550'), "didn't match 550")
    end
    makepath(ftp, 'first/child')
    ftp.chdir('first')
    assert(folder_exists?(ftp, 'child'), "should have existed")
    ftp.chdir(top)
    ftp.chdir('first')
    ftp.rmdir('child')
    ftp.chdir(top)
    assert(! folder_exists?(ftp, 'first/child'), "should have been deleted")
    ftp.close
  end

  def makepath(ftp, path)
    save = ftp.pwd
    Pathname.new(path).each_filename { |e| make_dir_safely(ftp, e) }
    ftp.chdir(save)
  end

  def make_dir_safely(ftp, name)
    name = name.to_s
    # puts "at #{ftp.pwd} making #{name}"
    ftp.mkdir(name) unless folder_exists?(ftp,name)
    ftp.chdir(name)
  end

  def folder_exists?(ftp, f)
    return ftp.list.any? { |item| item.match('^d.* ' + f)}
  end
end

Tozier’s not here yet for some reason. I think before I do anything else, I’ll try extracting a little FTP class, so that I can put these methods like makepath somewhere where they belong. That should be “easy”. Hold my beer.

In that way that he has, Tozier arrives. He suggests that our Publisher object already has some functionality in it, and maybe we should build from this test, into that, so that we don’t have to duplicate effort. I’m OK enough with that. Here’s Publisher:

# Publisher

require 'net/ftp'
require '/Users/ron/programming/passwords.rb'

class Publisher
  def initialize(source_root, remote_working_folder, host, mode)
    #of interest to us:
    #  source root (a full path on local computer)
    #  remote working folder (relative to what FTP system provides)
    @source_root = source_root
    @remote_working_folder = remote_working_folder # relative to ftp root
    user = Object.const_get('Passwords::' + mode + '_USER')
    password = Object.const_get('Passwords::' + mode + '_PASSWORD')
    @ftp = Net::FTP.new(host, user, password)
    @ftp_root = @ftp.pwd
    @ftp.chdir(@remote_working_folder.to_s)
  end

  def publish(content_pathname)
    target_path = content_pathname.dirname
    @ftp.mkdir(target_path.to_s) unless path_exists?(target_path)
    @ftp.chdir(target_path.to_s)
    @ftp.putbinaryfile((@source_root + content_pathname).to_s)
    @ftp.chdir(@ftp_root)
    @ftp.chdir(@remote_working_folder.to_s)
  end

  def path_exists?(path)
    return true if path.to_s == '.' 
    parent = path.parent
    base = path.basename
    things = @ftp.list(parent.to_s)
    things.any? { |name| name.match(base.to_s) }
  end

  def pwd
    return @ftp.pwd
  end

  def close
    @ftp.close
  end

end

Let’s try something simple and make it use Publisher.

Arrgh, no. This test doesn’t want to test two different objects. We could write a new test, referring to Publisher, and move things into it one by one, or we could slam this one to refer to Publisher and deal with all of our 7 assertions failing. We agree to start with a new test. And then we agree that since we’re testing Publisher already, we’ll enhance that test rather than build new tests into this morning’s experiment.

Here’s what we’re starting with:

require "minitest/autorun"
require "pathname"
require "date"
require "./site-builder"
require "./publisher"

class Test_Site_Builder  < Minitest::Test

  def setup
    @test_source_folder = Pathname.new('../site_builder_test_source')
    @test_jekyll_output_folder = 
      Pathname.new('/Users/ron/programming/site_builder_jekyll_output')
    @ftp_start = Pathname.new('/Users/ron')
    @remote_working_folder = 
      Pathname.new('programming/site_builder_ftp_root/httpdocs/')
  end

  def test_get_newer_files
    initialize_test_source
    sb = SiteBuilder.new(@test_source_folder, "")
    files = sb.get_newer_files
    assert_equal(3, files.size)
  end

  def test_corresponding_files
    initialize_test_source
    sb = SiteBuilder.new(@test_source_folder, @test_jekyll_output_folder)
    files = sb.get_newer_files
    files_to_move = sb.rendered_names

    assert_equal(files.length, files_to_move.length)
    path_to_find = @test_jekyll_output_folder + 'time_of_last_rendering'
    assert_includes(files_to_move, path_to_find)
    path_to_find = @test_jekyll_output_folder + 'find_me.html'
    assert_includes(files_to_move, path_to_find)
    path_to_find = @test_jekyll_output_folder + 'subfolder/subfolder_index.html'
    assert_includes(files_to_move, path_to_find)
  end

  def test_ftp_login
    publisher = create_test_publisher(@test_jekyll_output_folder, @remote_working_folder)
    assert_equal('/Users/ron/programming/site_builder_ftp_root/httpdocs', publisher.pwd)
    publisher.close
  end

  def test_ftp_a_root_file
    clear_output_folders
    put_file_in_root
    source_root = @test_jekyll_output_folder
    published_root = @ftp_start + @remote_working_folder
    publisher = create_test_publisher(
      source_root, @remote_working_folder)
    pathname_of_file_to_move = Pathname.new('strange_index.html')
    publisher.publish(pathname_of_file_to_move)
    full_pathname_of_published_file = published_root + pathname_of_file_to_move
    assert(full_pathname_of_published_file.exist?, "file didn't get created")
    publisher.close
  end

  def test_ftp_non_root_files
    clear_output_folders
    put_files_in_subfolder
    source_root = @test_jekyll_output_folder
    published_root = @ftp_start + @remote_working_folder
    publisher = create_test_publisher(
      source_root, @remote_working_folder)

    pathname_of_file_to_move = Pathname.new('sub_folder/substrange_index.html')
    publisher.publish(pathname_of_file_to_move)
    pathname_of_second_file_to_move = Pathname.new('sub_folder/secondstrange_index.html')
    publisher.publish(pathname_of_second_file_to_move)
    pathname_of_third_file_to_move = Pathname.new('sub_folder/child/childindex.html')
    publisher.publish(pathname_of_third_file_to_move)

    full_pathname_of_published_file =
      published_root + pathname_of_file_to_move
    assert(full_pathname_of_published_file.exist?, "file didn't get created")
    full_pathname_of_second_published_file = 
      published_root + pathname_of_second_file_to_move
    assert(full_pathname_of_second_published_file.exist?, "second file didn't get created")
    full_pathname_of_third_published_file = 
      published_root + pathname_of_third_file_to_move
    assert(full_pathname_of_third_published_file.exist?, "third file didn't get created")
    
    publisher.close
  end

  def create_test_publisher(source_root, remote_working_folder)
    host = 'localhost'
    mode = 'TEST'
    Publisher.new(source_root, remote_working_folder, host, mode)
  end

  def clear_output_folders
    @test_jekyll_output_folder.rmtree
    @test_jekyll_output_folder.mkdir
    to_remove = @ftp_start + @remote_working_folder
    to_remove.rmtree
    to_remove.mkdir
  end

  def put_file_in_root
    new_file = @test_jekyll_output_folder + 'strange_index.html'
    new_file.write('invalid html')
  end

  def put_files_in_subfolder
    new_file = @test_jekyll_output_folder + 'sub_folder/substrange_index.html'
    new_file.dirname.mkpath
    new_file.write('invalid html')
    second_file = @test_jekyll_output_folder + 'sub_folder/secondstrange_index.html'
    second_file.write('invalid html')
    third_file = @test_jekyll_output_folder + 'sub_folder/child/childindex.html'
    third_file.dirname.mkpath
    third_file.write('invalid html')
  end

  # def test_ftp
  #   initialize_test_source
  #   initialize_test_jekyll_output
  #   initialize_ftp_output_folder
  #   ftp = build_ftp_connection(host,target_folder)
  #   sb = SiteBuilder.new(
  #     @test_source_folder, 
  #     @test_jekyll_output_folder,
  #     ftp)
  #   sb.move_files
  #   assert_files_are_moved
  # end

  # utilities

  def initialize_test_source
    before = DateTime.new(2017,10,1,5,0,0)
    as_of = DateTime.new(2017,10,1,6,0,0)
    after = DateTime.new(2017,10,1,7,0,0)
    FileUtils.rm_rf(@test_source_folder)
    FileUtils.mkdir(@test_source_folder)
    FileUtils.mkdir(@test_source_folder + "subfolder")
    make_file(@test_source_folder, 'time_of_last_rendering', as_of)
    make_file(@test_source_folder, 'find_me.md', after)
    make_file(@test_source_folder, 'unfound.md', before)
    make_file(@test_source_folder, 'subfolder/subfolder_index.md', after)
    make_file(@test_source_folder, 'subfolder/not_found_index.md', before)
  end

  def make_file(base, path, date_time)
    name = base + path
    FileUtils.touch name, :mtime => date_time.to_time
  end

end

We decide to check the ftp_root of publisher, more to document it than for any other reason, and we do it in this existing test:

  def test_ftp_login
    publisher = create_test_publisher(@test_jekyll_output_folder, @remote_working_folder)
    assert_equal('/Users/ron', publisher.ftp_root)
    assert_equal('/Users/ron/programming/site_builder_ftp_root/httpdocs', publisher.pwd)
    publisher.close
  end

Adding attr_accessor :ftp_root to Publisher makes that run. What’s next?

    ftp.chdir(path)
    top = ftp.pwd
    assert_equal('/Users/ron/Dropbox/ftp-experiments', ftp.pwd)

This is already implied by our test above. The next bit checks whether folders exist or not:

    assert(folder_exists?(ftp, 'first'))
    assert(! folder_exists?(ftp,'nowhere'))

We need to translate this into the form of our target folders in the Publisher test. We haven’t really set those up with code yet: our tests clear folders but don’t set up anything below. That’s here:

  def clear_output_folders
    @test_jekyll_output_folder.rmtree
    @test_jekyll_output_folder.mkdir
    to_remove = @ftp_start + @remote_working_folder
    to_remove.rmtree
    to_remove.mkdir
  end

We should first move this into our setup, and then extend it as needed for further tests. We do that and return to writing this test:

  def test_can_find_existing_subfolder
    publisher = create_test_publisher(@test_jekyll_output_folder, @remote_working_folder)

  end

We need to have an existing folder to check. We’ll put it into clear_output_folders. Tozier notes that we should split that method up into ones that deal with the Jekyll output and the FTP. Well, actually he says something that makes me think that. We babble a while and then decide to write a failing test:

  def test_can_find_existing_subfolder
    publisher = create_test_publisher(@test_jekyll_output_folder, @remote_working_folder)
    assert(publisher.exists?('subfolder'), 'did not find supfolder')
  end

This fails because Publisher doesn’t know how to test for existence (and because the folder isn’t there anyway). So, we have in this morning’s test (and in some earlier code):

  def folder_exists?(ftp, f)
    return ftp.list.any? { |item| item.match('^d.* ' + f)}
  end

We can fit this into Publisher … except that we notice that we already have this:

  def path_exists?(path)
    return true if path.to_s == '.' 
    parent = path.parent
    base = path.basename
    things = @ftp.list(parent.to_s)
    things.any? { |name| name.match(base.to_s) }
  end

This does a similar thing. We could use it. I’m not certain whether FTP will list into a subfolder, which this code is assuming. In addition, this code will look for anything. The code this morning will look specifically for a folder under the current little yellow man. I’m going to use that and call it folder_exists_here?.

  def test_can_find_existing_subfolder
    publisher = create_test_publisher(@test_jekyll_output_folder, @remote_working_folder)
    assert(publisher.folder_exists_here?('subfolder'), 'did not find supfolder')
  end

And …

  def folder_exists_here?(folder)
    return @ftp.list.any? { |item| item.match('^d.* ' + folder)}
  end

This fails, pointing out the misspelling of subfolder … and that the folder isn’t there. Let’s put it there and see if maybe we can succeed. And we can:

  def test_can_find_existing_subfolder
    publisher = create_test_publisher(@test_jekyll_output_folder, @remote_working_folder)
    path = @ftp_start + @remote_working_folder + 'subfolder'
    path.mkdir
    assert(publisher.folder_exists_here?('subfolder'), 'did not find subfolder')
  end

OK we made the folder and the folder_exists_here method works.

Along the way, Tozier shows me that we could have written:

path = @ftp_start / @remote_working_folder / 'subfolder'

I consider that to be an abomination upon the land and refuse to use it, partly because it confuses Sublime’s Ruby parser, but mostly because it’s just too damn cute.

I note that something odd has happened in my mind. I came here planning to learn about FTP and how to build a convenient and compact FTP object. Now I’m thinking mostly about how to make Publisher work. These are not the same things. I’m not sure I’m happy about that but we’re here now. Sunk cost, anyone?

We can extend to testing for things that don’t exist, which is actually a useful test to show that our method isn’t too optimistic.

  def test_cannot_find_missing_subfolder
    publisher = create_test_publisher(@test_jekyll_output_folder, @remote_working_folder)
    assert(! publisher.folder_exists_here?('absent'), 'did not find subfolder')
  end

That works. Tozier proposes testing whether we can list down a nest to find a sub-sub-folder. I’m curious, so here we go:

  def test_can_find_sub_sub_folder
    publisher = create_test_publisher(@test_jekyll_output_folder, @remote_working_folder)
    path = @ftp_start + @remote_working_folder + 'sub/subsub/subsubsub'
    path.mkpath
    assert(publisher.folder_exists_here?('sub/subsub/subsubsub'), 'did not find subetcfolder')
  end

This fails given our current implementation of folder_exists_here. Tozier thinks we can make him better than he was before … he’s thinking we could “just” chdir to the folder in question and if that fails, it didn’t exist. This is true and my experiment this morning fielded a similar exception.

Is this too tricky, and too ugly, to do? Tozier has talked himself out of it, wanting to follow my thought of looping down the path one item at a time. Meanwhile, I want to follow his idea and see how ugly it is.

  def folder_exists_here?(path_string)
    begin
      @ftp.chdir(path_string)
    rescue Exception => e
      return false
    end
    return true
  end

This makes the test run. The exception is a bit heavy duty, so we need to narrow that down as I did this morning. First we’ll print it to see what it is. It’s “Net::FTPPermError: 550 absent: No such file or directory.” We revise to:

  def folder_exists_here?(path_string)
    begin
      @ftp.chdir(path_string)
    rescue Net::FTPPermError => e
      msg = e.message
      return false if msg.match('550')
      raise
    end
    return true
  end

The raise reraises the current exception in case we don’t like it. Thanks stackoverflow.

Well, this is useful. One question now is whether we still need the other exists method but I think we may. Another question is what, if anything, we want to import from this morning’s test. Or, what else might we want to do?

The next thing in this morning’s work is to make a path. The code remaining tested both making and removing:

    makepath(ftp, 'first/child')
    ftp.chdir('first')
    assert(folder_exists?(ftp, 'child'), "should have existed")
    ftp.chdir(top)
    ftp.chdir('first')
    ftp.rmdir('child')
    ftp.chdir(top)
    assert(! folder_exists?(ftp, 'first/child'), "should have been deleted")

We could write a test in Publisher to create some nest. It won’t be quite like this one, and I hope it isn’t quite like Toziers sub/sub, but maybe it should be … the test would mirror the previous one nicely.

And I’m done for the day … See you next time!