I wonder what we could do about comments. They’re tricky.

Codea has two kinds of comments. One kind is easy. Everything from -- to the end of the line is ignored. The other kind, well, they start with --[[ and end with the matching ]]. However, you can include equal signs in there, so that you can do nesting of comments or strings or I have no idea why you’d do this to be honest.

Anyway …

    print(1)
    --[=[
    print(2)
    --[[
    print(3)
    --]=]
    print(4)
    --]]

The above prints 1, and 4. However …

    print(1)
    -- [=[
    print(2)
    --[[
    print(3)
    --]=]
    print(4)
    --]]

That prints 1 and 2 because I spaced after the two dashes on the first comment, so it it no longer a block comment.

Our mission today is to skip comments. Let’s begin by making some new tests, My plan for comments is to remove them from the text before we even try to run our patterns. We’ll use the gsub function to turn them into nothing.

It took me a while to get this to work:

        _:test("Remove simple comments", function()
            local test = [[
kept
-- this is removed
kept-- this is removed also
kept
]]
            local pat = "%-%-.-\n"
            local ans = test:gsub(pat,"\n")
            _:expect(ans).is("kept\n\nkept\nkept\n")
        end)
    end)

Nothing exciting, just my usual slow progress with patterns. Anyway, that’s the easy one.

Lets try the other kind.

        _:test("Remove [[ comments", function()
local test = [=[
kept
kept--[[ this is removed
and this is also removed
--]]adjacent
kept
--[[ this is removed ]]
kept
]=]
            local pat = "%-%-%[%[.-%]%]"
            local ans = test:gsub(pat,"")
            _:expect(ans).is("kept\nkeptadjacent\nkept\n\nkept\n")
        end)

That actually worked. Note that I had to use the ‘[=[’ long string form, to avoid conflict with the comment square brackets, which will be perceived as terminating the string otherwise.

Why anyone would use this notation continues to elude me.

Now let’s see if we can insert the equal sign trick. I think we can.

        _:test("remove [=[ comments", function()
local test = [=[
kept1
kept2--[==[ this is removed
and this is also removed
--]==]adjacent
kept3
--[==[this is removed]==]
kept4
]=]
            local pat = "%-%-%[(=*)%[.-%]%1%]"
            local ans = test:gsub(pat,"")
            ans = ans:gsub("\n", ".")
            _:expect(ans).is("kept1.kept2adjacent.kept3..kept4.")
        end)

I had some trouble with this one because I left the one now called kept3 out by accident. That’s why I substituted the dot for newline and put in the numbers. This test runs.

That last pattern is the good one for the double square bracket kind of comments. I’ll paste it back into the second test.

I should also explain how it works.

Note the captured equal signs (=*) inside the left square brackets. Inside the right square brackets we see %1. That means “capture number 1”, so it requires the exact characters matched in between the [[. Neat, huh?

I’ll add those to the PatternDictionary and use them from there, to close the loop.

Commit: created comment-removing patterns.

Now I think we’d like to remove all comments from pages before we process them. Let’s have a look.

function ProjectDescription:init(projectName)
    self.projectName = projectName
    self.tabs = self:getProjectTabs(projectName)
    self.methodCollection = self:getMethods()
    self.classCollection = self:getClasses()
end

We’d like to be able to strip the comments right here and then run all the various get things. However, as it’s written now, we read the tabs multiple times:

function ProjectDescription:getClasses()
    local classes = {}
    for _i,tab in ipairs(self.tabs) do
        self:addClassesFromTab(tab,classes)
    end
    return ClassCollection(classes)
end

function ProjectDescription:getMethods()
    local methods = {}
    for _i,tab in ipairs(self.tabs) do
        self:addMethodsFromTab(tab, methods)
    end
    return MethodCollection(methods)
end

So we need to reorganize this class a bit. We now have our new MatchMaker object, so probably we should rewrite this test to use that. It’s set up to process tab contents:

function MatchMaker:process(tabContents)
    local matches = tabContents:gmatch(self.pattern)
    for cap1, cap2, cap3 in matches do
        local instance = self.defn(cap1,cap2,cap3)
        self.coll:add(instance)
    end
end

If we redo ProjectDescription correctly, to use the new MathMakers, we should be able to get the test running again quickly.

That goes fairly well, once I get over typing classMaker when I mean classMatcher. Changes were simple:

function ProjectDescription:init(projectName)
    self.projectName = projectName
    self.tabs = self:getProjectTabs(projectName)
    self.classMatcher = MatchMaker:classDefinition()
    self.methodMatcher = MatchMaker:methodDefinition()
    for _i,tab in ipairs(self.tabs) do
        local cont = tab:contents()
        self.classMatcher:process(cont)
        self.methodMatcher:process(cont)
    end
    self.classCollection = self.classMatcher:collection()
    self.methodCollection = self.methodMatcher:collection()
end

I guess all we want is the member collections that the Matchers have made for us so let’s make the other stuff local.

function ProjectDescription:init(projectName)
    self.projectName = projectName
    self.tabs = self:getProjectTabs(projectName)
    local classMatcher = MatchMaker:classDefinition()
    local methodMatcher = MatchMaker:methodDefinition()
    for _i,tab in ipairs(self.tabs) do
        local cont = tab:contents()
        classMatcher:process(cont)
        methodMatcher:process(cont)
    end
    self.classCollection = classMatcher:collection()
    self.methodCollection = methodMatcher:collection()
end

Commit: convert ProjectDescription to use MatchMakers.

Now we’ll put some comments in the Sample and break the tests.

-- SampleCode
-- RJ 20211021

SampleCode = class()

-- FakeClass = class()

--[[
FakeClass2 = class()
--]]

function SampleCode:init(x)
    -- you can accept and set parameters here
    self.x = x
end

-- function FakeClass:fakeMethod()

--[=[
function FakeClass:fakeMethod2()
--]=]

function SampleCode:draw()
    -- Codea does not automatically call this method
end

function SampleCode:touched(touch)
    -- Codea does not automatically call this method
end

SubclassOfSample = class(SampleCode)

function SubclassOfSample:init(ignored)
    
end

function SampleCode:extraMethod(a,b,c)
    -- blah
end

I expect this to produce too many classes and methods in the report. And other detail tests will also fail.

6: Find all classes  -- Actual: FakeClass, Expected: SubclassOfSample
2: ProjectDescription methods  -- Actual: 7, Expected: 5
3: ProjectDescription classes  -- Actual: 4, Expected: 2
4: ProjectDescription classReport  -- Actual: Classes

    SampleCode
    FakeClass
    FakeClass2
    SubclassOfSample(SampleCode)
, Expected: Classes

    SampleCode
    SubclassOfSample(SampleCode)

1: MatchMaker finds classes  -- Actual: 4, Expected: 2
1: MatchMaker finds classes  -- Actual:     SampleCode
    FakeClass
    FakeClass2
    SubclassOfSample(SampleCode)
, Expected:     SampleCode
    SubclassOfSample(SampleCode)

1: MatchMaker finds classes  -- Actual:     SampleCode
    FakeClass
    FakeClass2
    SubclassOfSample(SampleCode)
    Argle
, Expected:     SampleCode
    SubclassOfSample(SampleCode)
    Argle

2: MatchMaker finds methods  -- Actual:     SampleCode:init(x)
    FakeClass:fakeMethod()
    FakeClass:fakeMethod2()
    SampleCode:draw()
    SampleCode:touched(touch)
    SubclassOfSample:init(ignored)
    SampleCode:extraMethod(a,b,c)
, Expected:     SampleCode:init(x)
    SampleCode:draw()
    SampleCode:touched(touch)
    SubclassOfSample:init(ignored)
    SampleCode:extraMethod(a,b,c)

Looks like we have a lot of comment removing to do. We could remove some of these tests, since the latest ProjectDescription test covers everything, I think.

Let’s fix it first, in any case.

I remove this test, which is checking a comment:

        _:test("Read SampleCode tab from this project", function()
            local pattern = "%-%-%s*(SampleCode)"
            local start,stop,capture = tab:find(pattern)
            _:expect(start).is(1)
            _:expect(stop).is(13)
            _:expect(capture).is("SampleCode")
        end)

I implement this function:

function stripComments(aString)
    local result = aString:gsub(PatternDictionary:commentMulti(),"")
    result = result:gsub(PatternDictionary:commentSingle(), "\n")
    return result
end

And call it as necessary in the tests. They are all green.

We have successfully removed comments from our input, and are still producing the right outputs. Life is good.

Commit: Comments of both types are ignored in ProjectStatistics.

Let’s sum up.

Summary

Today went surprisingly well. The comment stripping patterns were easy enough to do, with the nice %1 matching capability of the Lua patterns. Changing the ProjectDescription to process each tab just once was straightforward, and so was converting the tests all to remove comments from our sample project

One thing that is a bit odd is the stripComments function, which is a free-hanging function rather than a method on some object. We could certainly build something.

Perhaps tab:contents should strip comments automatically unless told otherwise. That might be a nice way of doing it.

We’ll save that for next time. For now, I’m outa here.

I’ll see both of you next time, I hope.