The New New Product Development Game for iPad

After the discussion last time, we’ve decided to start on a clean slate. Let me be right up front here and acknowledge that my usual position is never to rewrite a product. But I’m talking there about something we’ve invested many person-years in, not something where we’ve worked for just a few days or weeks. We’ve learned a lot in our few weeks, and while maybe we could refactor it into our existing code, we’re going to try starting over. Remember, this is The Book of Erors, so whatever happens, it’s all good.

As we begin, Tozier points out that we have a history of looking at the wrong files and folders already and that we should be extra careful not to do that as it leads to confusion. I’m thinking we should rename most of our top-level tests and classes.

OK, let’s build a new place to stand. We create a new git repo, rj-com-site-builder, and with non-zero effort, connect it to GitHub. Now, I suppose, we need a test.

require "minitest/autorun"
#require "./site-builder"

class Test_Site_Builder  < Minitest::Test

end

This test runs. We’re hooked up. We discuss and agree that our new site-builder class will be in a separate file (and presumably set up to run as main by the time we’re done.)

Now what?

Tozier suggests making a list of the files to be moved, since we most recently worked on that. I think I agree, but I’d like to sketch here the overall flow of the app first.

# Here's fake Ruby for the site-builder

while TRUE {
    wait(somePeriodOfTime) # does Ruby spin CPU here? if so cron
    files = list_files_newer_than(time_of_last_rendering)
    if ( ! files.empty? ) {
        run_jekyll()
        move_corresponding_files(files, various_other_info)
        save_time_of_last_rendering()
    }
}

Subject to some details to be worked out, this is roughly what we want this new app to do. Assuming that Ruby’s wait didn’t spin the CPU, we could use this. Tozier wants to assume we use cron, because he finds it weird and scary that Ruby would be sitting there running forever. I’m OK with that and if we do that then the first line above is commented out. So we can use this sketch.

This makes us think that getting the list of newer files is a good starting point. Of course we will need to figure out what we mean by time_of_last_rendering along the way.

We “decided” last time to use File#mtime, the modification time, as our time hack. Or probably PathName#mtime, anyway mtime. Where and how shall we represent our time_of_last_rendering? Probably just a file, probably in the site root. But should we write our tests to look at the real site source, or build a test site? Probably a test site will be better.

Whee, this means we’ll have a big list of folder names that drive our objects. I think I’ve seen this road before. Well, we’re here to do a better job so let’s get started … and do a better job.

We’ll start by putting operational code in our test file, and move it out as soon as possible into the app source. But let’s start. After a bit more futzing than one could be proud of, we get this:

require "minitest/autorun"
require "pathname"

class Test_Site_Builder  < Minitest::Test

  def test_get_newer_files
    source_folder = Pathname.new('../site_builder_test_source')
    last_time_file = source_folder + 'time_of_last_rendering'
    last_time = last_time_file.mtime
    files = get_newer_files(source_folder, last_time)
    puts files
    assert_equal(2, files.size)
  end

  # Code to be moved to app 

  def get_newer_files(source_folder, last_time)
    source_files_and_folders = Pathname.glob(source_folder + '**/*')
    return source_files_and_folders.select do |f| 
      f.file? && f.mtime > last_time
    end
  end

end

We manually created this folder and file structure:

• site_builder_test_source
  • find_me.md
  • subfolder
    • subfolder_index.md

We’re filtering only files, so we expect to see only find_me.md and subfolder_index.md and in fact we do. Our test does not check for them yet but we looked.

Our next step will be to make the folders and files inside our test, and perhaps (or perhaps not) to make some more explicit tests. We’re happy, though, that this scheme is going to work.

But for now, we’ll break. Very short day today. That’s how it goes sometimes.

On My Own …

I decided today to enhance the tests to create the test folder in setup with no operational changes to our tiny bit of “application” code. The result was this:

require "minitest/autorun"
require "pathname"
require "date"

class Test_Site_Builder  < Minitest::Test

  def setup
    @test_folder = Pathname.new('../site_builder_test_source')
  end

  def test_get_newer_files
    initialize_test_source
    source_folder = @test_folder
    last_time_file = source_folder + 'time_of_last_rendering'
    last_time = last_time_file.mtime
    files = get_newer_files(source_folder, last_time)
    puts files
    assert_equal(2, files.size)
  end

  # utilities

  def initialize_test_source
    before = DateTime.new(2017,10,1,5,0,0)
    as_of = DateTime.new(2017,10,1,6,0,0)
    after = DateTime.new(2017,10,1,7,0,0)
    FileUtils.rm_rf(@test_folder)
    FileUtils.mkdir(@test_folder)
    FileUtils.mkdir(@test_folder + "subfolder")
    make_file(@test_folder, 'time_of_last_rendering', as_of)
    make_file(@test_folder, 'find_me.md', after)
    make_file(@test_folder, 'unfound.md', before)
    make_file(@test_folder, 'subfolder/subfolder_index.md', after)
    make_file(@test_folder, 'subfolder/not_found_index.md', before)
  end

  def make_file(base, path, date_time)
    name = base + path
    FileUtils.touch name, :mtime => date_time.to_time
  end

  # Code to be moved to app 

  def get_newer_files(source_folder, last_time)
    source_files_and_folders = Pathname.glob(source_folder + '**/*')
    return source_files_and_folders.select do |f| 
      f.file? && f.mtime > last_time
    end
  end

end

This runs our one assertion as before, with no failures and no errors. I’ve not built a teardown yet, which let me check manually that the files were set up as intended. I decided not to write a meta-test to test that …

For now, this step will do.

Resuming after Tozier arrives …

Tozier approves of my initialize code to create the folders. And he asks about the test above:

  def get_newer_files(source_folder, last_time)
    source_files_and_folders = Pathname.glob(source_folder + '**/*')
    return source_files_and_folders.select do |f| 
      f.file? && f.mtime > last_time
    end
  end

Should it say f.mtime >= last_time? We think that it should, and in addition, upon starting this job, maybe we need to set last_time as soon as we start. And Chet¹ asks whether we should actually build all files that are close to last_time but before it, just in case something came in that our finder doesn’t see. This would be harmless, because rebuilding a page doesn’t do any harm, while if we ever miss a file, we’ll never build it if its date is (sufficiently) less than our build time.

After discussion we agree that we’ll include files in the build that are close to the last time built, but before it. (This is hard to express.) We’ll do that by providing a different last_date in the call to this function. At least that’s my plan. I don’t see building a random constant into this low-level function.

We’re confident that, given the right values, our little get_newer_files function gets the right files. We need a new test … I think. To decide what to do, we revise our sketch:

# Here's fake Ruby for the site-builder

while TRUE {
    wait(somePeriodOfTime) # does Ruby spin CPU here? if so cron
    files = list_files_newer_than(time_of_last_rendering)
    save_new_time_of_last_rendering() #moved up
    if ( ! files.empty? ) {
        run_jekyll()
        move_corresponding_files(files, various_other_info)
    }
}

So it seems like we should write a test for move_corresponding_files and make that work, then build our new site-building program, moving the code out of out test file and into the real thing. Anyway, we need a test for FTP moving.

There are three “locations” of interest: the site source, the built site, and the FTP destination. Our mover takes the names from the site source, finds the corresponding names in the built site, and moves them to the same names on the production site. This makes me ask whether the test we’re about to write should run Jekyll.

Tozier says we should just return the list of corresponding names and not do a move at all. Well, this is a smaller test, so I’m all for it.

  def test_corresponding_files
    # set up some files, maybe same as other test
    # find the new ones
    # produce the transformed collection and check it
  end

That’s our sketch.

  def test_corresponding_files
    initialize_test_source
    source_folder = @test_folder
    last_time_file = source_folder + 'time_of_last_rendering'
    last_time = last_time_file.mtime
    files = get_newer_files(source_folder, last_time)
    files_to_move = transform_names(files)
    assert_includes(files_to_move, Pathname.new('../site_builder_test_source/find_me.md'))
    # produce the transformed collection and check it
  end

  def transform_names(names)
    return names
  end

This test succeeds, because we asked for .md. We really want our transform to produce .html. And is that the path we want now? I think not. We want the path to the file to be in the rendered site folder. So we improve the test and make it run …

We hammer a bit and make our test pass with this:

  def transform_names(names)
    return names.collect { |f| transform_name(f) }
  end

  def transform_name(file)
    relative_path = file.relative_path_from(@test_source_folder)
    new_base = relative_path.to_s.gsub(/\.md$/,".html")
    @test_jekyll_output_folder + new_base
  end

This bit of arcanity in transform_name “just” strips the source folder’s name from the file, substitutes ‘html’ for ‘md’, and prepends the output folder’s path to it, producing the desired new pathname.

We consider this code to be less clear than it might be. What to do …

  def transform_name(pathname)
    filename_relative_to_home = pathname.relative_path_from(@test_source_folder)
    transformed_filename = substitute_html(filename_relative_to_home)
    @test_jekyll_output_folder + transformed_filename
  end

  def substitute_html(pathname)
    pathname.to_s.gsub(/\.md$/,".html")
  end

This relies on the fact that Pathname acts a lot like a string, so you can add a string to a Pathname and get a new Pathname back. We decide to let this slide.

It’s odd, though, that even with Pathname seeming “better”, we’re still slipping back and forth between strings and Pathnames a lot. But we’ve pretty much cracked the issue of the Dir object having a built-in implied basically unknowable base folder.² Our base folders are currently the two member variables of our test class:

class Test_Site_Builder  < Minitest::Test

  def setup
    @test_source_folder = Pathname.new('../site_builder_test_source')
    @test_jekyll_output_folder = Pathname.new('../site_builder_jekyll_output')
  end

  ...

Our “real” class will have some similar way of finding out what folders it’s supposed to use. That remains to be figured out.

Anyway our proposed test runs: we can create a set of Pathname objects that represent the files that need to be FTPed up to the web site. We need to enhance this test to be more certain of its behavior. Let’s see what we can gin up.

  def test_corresponding_files
    initialize_test_source
    source_folder = @test_source_folder
    last_time_file = source_folder + 'time_of_last_rendering'
    last_time = last_time_file.mtime
    files = get_newer_files(source_folder, last_time)
    files_to_move = transform_names(files)
    assert_equal(files.length, files_to_move.length)
    path_to_find = @test_jekyll_output_folder + 'time_of_last_rendering'
    assert_includes(files_to_move, path_to_find)
    path_to_find = @test_jekyll_output_folder + 'find_me.html'
    assert_includes(files_to_move, path_to_find)
    path_to_find = @test_jekyll_output_folder + 'subfolder/subfolder_index.html'
    assert_includes(files_to_move, path_to_find)
  end

We make the test a bit more comprehensive. Still works. T points out that in our transform, we could pull the pathname apart to get the extension and then put the transformed extension back. But our gsub does say /.md$/, so it will only get the final extension anyway. Seems close enough for today.

At this point we have a list of Pathnames representing files that we wish to FTP to the site. We still have to write the FTP code and do the safe folder creation, which we’ve done before. I’m inclined to build a test for this, produce whatever functions inside the test we need, and only then begin to build up the real application file and classes.

And that’s for another time, as our time is coming to a close. Here’s the whole program for the retentive among us:

require "minitest/autorun"
require "pathname"
require "date"

class Test_Site_Builder  < Minitest::Test

  def setup
    @test_source_folder = Pathname.new('../site_builder_test_source')
    @test_jekyll_output_folder = Pathname.new('../site_builder_jekyll_output')
  end

  def test_get_newer_files
    initialize_test_source
    source_folder = @test_source_folder
    last_time_file = source_folder + 'time_of_last_rendering'
    last_time = last_time_file.mtime
    files = get_newer_files(source_folder, last_time)
    puts files
    assert_equal(3, files.size)
  end

  def test_corresponding_files
    initialize_test_source
    source_folder = @test_source_folder
    last_time_file = source_folder + 'time_of_last_rendering'
    last_time = last_time_file.mtime
    files = get_newer_files(source_folder, last_time)
    files_to_move = transform_names(files)
    assert_equal(files.length, files_to_move.length)
    path_to_find = @test_jekyll_output_folder + 'time_of_last_rendering'
    assert_includes(files_to_move, path_to_find)
    path_to_find = @test_jekyll_output_folder + 'find_me.html'
    assert_includes(files_to_move, path_to_find)
    path_to_find = @test_jekyll_output_folder + 'subfolder/subfolder_index.html'
    assert_includes(files_to_move, path_to_find)
  end

  # utilities

  def initialize_test_source
    before = DateTime.new(2017,10,1,5,0,0)
    as_of = DateTime.new(2017,10,1,6,0,0)
    after = DateTime.new(2017,10,1,7,0,0)
    FileUtils.rm_rf(@test_source_folder)
    FileUtils.mkdir(@test_source_folder)
    FileUtils.mkdir(@test_source_folder + "subfolder")
    make_file(@test_source_folder, 'time_of_last_rendering', as_of)
    make_file(@test_source_folder, 'find_me.md', after)
    make_file(@test_source_folder, 'unfound.md', before)
    make_file(@test_source_folder, 'subfolder/subfolder_index.md', after)
    make_file(@test_source_folder, 'subfolder/not_found_index.md', before)
  end

  def make_file(base, path, date_time)
    name = base + path
    FileUtils.touch name, :mtime => date_time.to_time
  end

  # Code to be moved to app 

  def get_newer_files(source_folder, last_time)
    source_files_and_folders = Pathname.glob(source_folder + '**/*')
    return source_files_and_folders.select do |f| 
      f.file? && f.mtime >= last_time
    end
  end

  def transform_names(names)
    return names.collect { |f| transform_name(f) }
  end

  def transform_name(pathname)
    filename_relative_to_home = pathname.relative_path_from(@test_source_folder)
    transformed_filename = substitute_html(filename_relative_to_home)
    @test_jekyll_output_folder + transformed_filename
  end

  def substitute_html(pathname)
    pathname.to_s.gsub(/\.md$/,".html")
  end

end

New Day

It’s a new day here at the BAR³, and as soon as Tozier gets here, we’ll write another few lines of code. Well, after we catch up on recent events and refresh our memories of what we were doing Lo! these four days ago. I’ll start my review now. Don’t wander off, I’ll be right back.

One thing that concerns me a bit is that our get_newer_files function just returns a bare array of file names. Then we create a new array of file names, just like the other ones except with .md replaced by .html. We plan to write code that will run over this array and FTP the files from the Jekyll-generated site pages up to my website. We’ve done this before, and I think the biggest real problem facing us is that I’ll be tempted to skip our test that runs FTP internally on my laptop, and instead convince myself that it’s going to do the right thing, and turn it loose on my real site. That would almost certainly work, but it’s clearly dangerous to the point of madness, given the ramifications if (when?) it doesn’t work.

But my spidey object sense tells me that there wants to be an object that knows about the Jekyll source folder, the Jekyll output folder, the target folder on my site, and the FTP information, and that does the job of finding out what to do, and then subsequently doing it.

But “there isn’t much there, there”. Our whole process is something like

• see what’s new
• run Jekyll
• move output files corresponding to “what’s new” to my site

Arguably, the code we’re writing is the object I’m thinking about, or will be once we begin making a class out of it.

Digression

Why am I obsessing over this tiny program that anyone could write in a day or so, if not in a few hours? Why not crack it out and be done?

I’m glad you asked that. The reason is that – in my somewhat experienced opinion – the very same issues and topics arise in small programs as in large ones. Large programs, done well, are created by breaking down what they have to do into many small things, and then doing lots of small programs that add up to the big solution.

Therefore … the issues and topics we work with here, in this simple sandbox, equip us to think about and deal with the problems we encounter in the larger “real world” situation.

At least that’s my story. I will probably stick to it.

End Digression

All this makes me think that it’s about time to move our in-line code from our test file, out to an actual class that does the job we have in mind. I’ll kick that around with Tozier when he gets here.

… and he’s here. And we agree that what we’ll do next is create an application class and push the behavior from our tests into it.

This is not entirely as stupid as it may seem. When we’re figuring out ways to do things, working in IRB, or in tiny patches of code, or as methods inside our test case, lets us focus more easily on the behavior and less on the boilerplate. It’s time, however, for the boilerplate.

We begin with this giant leap:

# site builder

class SiteBuilder
end

Now let’s see about our first test:

  def test_get_newer_files
    initialize_test_source
    source_folder = @test_source_folder
    last_time_file = source_folder + 'time_of_last_rendering'
    last_time = last_time_file.mtime
    files = get_newer_files(source_folder, last_time)
    puts files
    assert_equal(3, files.size)
  end

We transform to this:

  def test_get_newer_files
    initialize_test_source
    sb = SiteBuilder.new(@test_source_folder)
    # last_time_file = source_folder + 'time_of_last_rendering'
    # last_time = last_time_file.mtime
    files = sb.get_newer_files
    puts files
    assert_equal(3, files.size)
  end

We’re assuming that SiteBuilder will absorb the source folder name, know on its own what the name of the time file is, and get the files. This test will fail. I think we’ll comment out the second test for now. We’ll look at it shortly.

We rather quickly get this:

# site builder

class SiteBuilder
  def initialize(source_folder)
    @source_folder = source_folder
  end

  def get_newer_files
    last_time_file = @source_folder + 'time_of_last_rendering'
    last_time = last_time_file.mtime
    source_files_and_folders = Pathname.glob(@source_folder + '**/*')
    return source_files_and_folders.select do |f| 
      f.file? && f.mtime >= last_time
    end
  end
end

Driven by this:

  def test_get_newer_files
    initialize_test_source
    sb = SiteBuilder.new(@test_source_folder)
    files = sb.get_newer_files
    puts files
    assert_equal(3, files.size)
  end

And the test runs. We decide we don’t need the puts any more, that was just there to comfort us.

But Ron! Aren’t you supposed to be doing TDD? Shouldn’t you be learning everything from your tests? Therefore, shouldn’t you have written a test to see if the files were right? Well, in a perfect world, with a perfect programmer, maybe. But really, there’s a difference between seeing a thing, and asking questions about the thing, and sometimes, I think it’s better just to look. You need to find your own way of doing things. My job here is to describe how I work, and to show you as honestly as I can what happens.

Now to make the second test run as well. It starts like this:

  def test_corresponding_files
    initialize_test_source
    source_folder = @test_source_folder
    last_time_file = source_folder + 'time_of_last_rendering'
    last_time = last_time_file.mtime
    files = get_newer_files(source_folder, last_time)
    files_to_move = transform_names(files)
    assert_equal(files.length, files_to_move.length)
    path_to_find = @test_jekyll_output_folder + 'time_of_last_rendering'
    assert_includes(files_to_move, path_to_find)
    path_to_find = @test_jekyll_output_folder + 'find_me.html'
    assert_includes(files_to_move, path_to_find)
    path_to_find = @test_jekyll_output_folder + 'subfolder/subfolder_index.html'
    assert_includes(files_to_move, path_to_find)
  end

We get this far:

  def rendered_names
    return get_newer_files.collect { |f| transform_name(f) }
  end

  def transform_name(pathname)
    filename_relative_to_home = pathname.relative_path_from(@source_folder)
    transformed_filename = substitute_html(filename_relative_to_home)
    @test_jekyll_output_folder + transformed_filename
  end

  def substitute_html(pathname)
    pathname.to_s.gsub(/\.md$/,".html")
  end

And realize that our transform_name method needs to know the name of the Jekyll output folder, which we haven’t provided in our constructor. So we provide that, then discover we needed to get the newer files in our second test, and wind up with this:

  def test_corresponding_files
    initialize_test_source
    sb = SiteBuilder.new(@test_source_folder, @test_jekyll_output_folder)
    files = sb.get_newer_files
    files_to_move = sb.rendered_names
    
    assert_equal(files.length, files_to_move.length)
    path_to_find = @test_jekyll_output_folder + 'time_of_last_rendering'
    assert_includes(files_to_move, path_to_find)
    path_to_find = @test_jekyll_output_folder + 'find_me.html'
    assert_includes(files_to_move, path_to_find)
    path_to_find = @test_jekyll_output_folder + 'subfolder/subfolder_index.html'
    assert_includes(files_to_move, path_to_find)
  end

And our new class, complete:

# site builder

class SiteBuilder
  def initialize(source_folder, jekyll_output_folder)
    @source_folder = source_folder
    @jekyll_output_folder = jekyll_output_folder
  end

  def get_newer_files
    last_time_file = @source_folder + 'time_of_last_rendering'
    last_time = last_time_file.mtime
    source_files_and_folders = Pathname.glob(@source_folder + '**/*')
    return source_files_and_folders.select do |f| 
      f.file? && f.mtime >= last_time
    end
  end

  def rendered_names
    return get_newer_files.collect { |f| transform_name(f) }
  end

  def transform_name(pathname)
    filename_relative_to_home = pathname.relative_path_from(@source_folder)
    transformed_filename = substitute_html(filename_relative_to_home)
    @jekyll_output_folder + transformed_filename
  end

  def substitute_html(pathname)
    pathname.to_s.gsub(/\.md$/,".html")
  end

end

I claim, correctly, that we don’t need our first test. Tozier agrees that it is implicitly covered by our second test but refuses to give it up, clinging to its tiny shred of history and confidence-building with a tenacity which can only be admired. “Don’t turn your back, Bill!”

Now we’re back to where we started, with cleaner and decently-factored code. We are, of course, ripping the innards out of our new little class, in order to test it, but that seems almost necessary. By that I mean “we can’t think of a better thing to do right now”.

Our mission now will be to work on the FTP side: moving these Jekyll output files up to my site.

Tozier falls into the slough of despond worrying about the fact that our current implementations have my real passwords stored in a file, and presently that file is actually inside our project. This means that there’s a danger that we’ll upload my passwords to the web site (which is only prevented because our current _config.yml happens to exclude .rb files), and a danger that we’ll put the passwords file into my GitHub repo, which can be prevented with .gitignore.

There are ways around this, such as storing the password file externally to the project, or even using system environment variables to contain them instead of a Ruby source file. (We found this idea on the Internet, we weren’t clever enough to think of it.) We don’t need to address this right now, but since we’re about to write code to reference the passwords, we could at least host them externally to our project and to Dropbox.

We need a test. We’re struggling among ourselves about what it should be. I have in mind something like this:

  def test_ftp
    # set up source folder
    # set up fake Jekyll output folder
    # set up FTP target folder
    sb = SiteBuilder.new(
      test_source_folder, 
      test_jekyll_output_folder,
      ftp_host,
      ftp_target_folder,
      password_prefix)
    sb.move_files
    assert_files_are_moved
  end

I don’t want to run Jekyll to write this test. (I may be just deferring the inevitable but I feel that we’ve solved all these problems before and I don’t want to solve them again.) However, with this test, we’re assuming the same basic solution as before, and Tozier points out that one could imagine a SiteBuilder that was given a live FTP connection rather than embedding all the information needed to create one. As imagined here, SiteBuilder seems a bit less cohesive than we’d like, because it has information about our structure from source to output to site, but also tricksy information about passwords and FTP creation. And with the prefix, it at least carries (but doesn’t think about) the test vs production status.

We decide to try it. We’ll create an FTP in the test and pass it in, deferring the question of how we make it work in production. This will surely drive us to create at least a main program for SiteBuilder but we know we need something there anyway. It has to run, after all.

So our new sketched test is:

  def test_ftp
    # set up source folder
    # set up fake Jekyll output folder
    # set up FTP target folder
    # create an FTP connected to FTP target folder
    sb = SiteBuilder.new(
      test_source_folder, 
      test_jekyll_output_folder,
      ftp)
    sb.move_files
    assert_files_are_moved
  end

OK, we have code we can borrow to do this, probably, so let’s try.

Yes, well. That didn’t happen. We looked at our old FTP code, which I’ll show you below, and we realized that what we did then was create a new FTP connection for each phase of our move. If we’re to pass in a single FTP, we need to resolve how to get our single FTP object moved to some deeply embedded folder, then back out to the top so as to move to another.

We did a little research and discovered two possible approaches. First, FTP has a pwd operation, so we could save and restore the old location, returning to a known state. In addition, Tozier discovered that the Pathname object can compute a relative path from anywhere to anywhere else in the tree, with lots of really neat ../../../ stuff. This is ugly as sin to look at but given that it works it would mean that we could set to a new FTP target folder using relative path from “wherever you are now” to “wherever I want to go”.

These two approaches are essentially equivalent, in that the action will happen somewhere deep inside the FTP activity. Save/restore, or compute offset from where we are, carry on (keeping calm of course).

We will go forward and write a test that really does FTP, along the lines of the sketches shown here, next time. Unless we don’t.

We’re done for the day. See you next time.