Codea Stats 2
Let’s do methods, and try to drive out some objects.
I think today I’ll devise a regex to recognize a method definition, and record them as associated with corresponding classes. Off the top of my um head, what will be needed? Something like these items:
- a regex recognizing a method definition
- perhaps its argument list?
- an object, maybe
ClassDefinitionto serve as a holder for the discovered methods of that class. - a collection, preferably smart rather than raw table, containing ClassDefinitions stored by name, to be looked up and given discovered methods.
One issue is that it’s possible that a method definition isn’t in the same tab as the class definition. That’s why I think we need a keyed collection of ClassDefinition, so that we can tuck the methods into the right class even if they show up in some odd order.
Another issue that comes to mind is that there are some local classes in some apps, even local classes that may have duplicate names. I think FakeTile is one of them. We may choose to do something about that.
My overall plan for this is to enhance the sample program with some method definitions to test, TDD out the regex and then the rest. I’m going to hang loose and let the problem and solution tell me what we need.
I suppose that sometime soon we’ll have to think about some kind of output. Maybe not today.
Let’s Go
I added some lines to the sample program:
SubclassOfSample = class(SampleCode)
function SubclassOfSample:init(ignored)
end
function SampleCode:extraMethod(a,b,c)
-- blah
end
There we have a method for our subclass, which didn’t have any methods before, followed by an out-of-place method for SampleCode. In the fullness of time, we’ll test that that shows up in the right place.
First the regex. Just because I’m thinking about it, I write the method first. So sue me.
function PatternDictionary:methodDefinition()
return "%s*function%s*(%w+):(%w+)%((.*?)%)"
end
Now let’s test to see if it remotely works.
_:test("Find a method", function()
local pattern = pd:methodDefinition()
start,stop,className,methodName, args = tab:find(pattern)
_:expect(className).is("SampleCode")
_:expect(methodName).is("init")
_:expect(args).is("x")
end)
Let’s run it, see what happens.
All three of those expects manage to fail, with nil as their return. I guess my regex is flawed. I’ll try paring it down, see if we can get something to come out. In aid of that, some printing in the test.
First print, every return is nil. Matched nothing. No surprise there. Let’s do check start though, expecting 1.
Next a trivial pattern:
function PatternDictionary:methodDefinition()
--return "%s*function%s*(%w+):(%w+)%((.*?)%)"
return "%s*function%s*"
end
This will hopefully give us something. (I wonder if colon character is special in our regex. That could be the problem. Patience. We’ll get there.
The test finds characters 51-61. This surprises me a bit, because there are 11 characters found. I wonder what they are?
Let’s print that too.
_:test("Find a method", function()
local pattern = pd:methodDefinition()
local start,stop,className,methodName, args = tab:find(pattern)
print("find method", start, stop, className, methodName, args)
_:expect(start).is(1)
local found = tab:sub(start,stop)
print(found)
_:expect(className).is("SampleCode")
_:expect(methodName).is("init")
_:expect(args).is("x")
end)
Prints “function” but of course I can’t see any spaces. Changing the print gives me this:
/
function /
We’re finding more whitespace than I had in mind. No biggie, we’re ignoring it anyway.
Let’s look to see if colon is a magic character in regex. Doesn’t seem to be. Extend the regex:
return "%s*function%s*(%w+):"
Test. Looks like it worked up to the name. Enhance the test for more clear output:
_:test("Find a method", function()
local pattern = pd:methodDefinition()
local start,stop,className,methodName, args = tab:find(pattern)
print("find method", start, stop, className, methodName, args)
_:expect(start, "start").is(51)
local found = tab:sub(start,stop)
print('/'..found..'/')
_:expect(className,"className").is("SampleCode")
_:expect(methodName,"methodName").is("init")
_:expect(args,"args").is("x")
end)
Run again:
find method 51 72 SampleCode nil nil
8: Find a method start -- OK
/
function SampleCode:/
8: Find a method className -- OK
8: Find a method methodName -- Actual: nil, Expected: init
8: Find a method args -- Actual: nil, Expected: x
That’s as I’d expect so far. Extend the regex.
return "%s*function%s*(%w+):(%w+)%("
This should get the method name right, and the print should include the left paren. Test. Works as expected. Hmm. I’m not really copying from my saved regex, I’m redoing it. This is pretty much how I always do regexes once my first guess fails: bit by bit.
return "%s*function%s*(%w+):(%w+)%(.*?%)"
I think this means, any character sequence, not greedy, followed by right paren. This is going to fail, because it is what I had before. (I looked after I wrote this.) Test. Yes, it fails. It comes back all nil. Ah. I’m wrong about the ?. That means match zero or one. Time for some Internet research.
Ah. Lua patterns (which aren’t really regexes) use - in place of * for a non-greedy match. So:
return "%s*function%s*(%w+):(%w+)%(.-%)"
I hope the test will run. Hope isn’t a strategy, but of course I’m not so sure as to actually expect it. Test:
8: Find a method args -- Actual: nil, Expected: x
Right. Didn’t capture it.
return "%s*function%s*(%w+):(%w+)%((.-)%)"
This one I expect to work. I could be wrong. Test. Test runs. I declare this pattern open for business. Clean up the test a bit.
_:test("Find a method", function()
local pattern = pd:methodDefinition()
local start,stop,className,methodName, args = tab:find(pattern)
_:expect(className,"className").is("SampleCode")
_:expect(methodName,"methodName").is("init")
_:expect(args,"args").is("x")
end)
Test again. Still good. Commit: define pattern to recognize a method definition.
Now let’s write the larger test that finds and checks them all, driving out whatever gmatch we wind up needing.
My vision for this program, I don’t know if I have made it clear, is that we’ll just scan each tab for our various patterns unconditionally, and sort out the results based on the information that comes back. That is, we’ll find all class definitions in the tab, then all method definitions, rather than looking for methods for that specific class.
Here’s a test against the sample tab we care about:
_:test("Tabs class", function()
local tabs = Tabs("SampleStatsProject")
local tab = tabs:at(2)
_:expect(tab:tabName()).is("SampleCode")
_:expect(tab:numberOfClasses()).is(2)
end)
We’ll want a similar test for methods. Let’s see how the Tabs class, and its friend Tab, works:
Tabs = class()
function Tabs:init(projectName)
self.projectName = projectName
local names = listProjectTabs(projectName)
local tabs = {}
for _i,name in ipairs(names) do
table.insert(tabs, Tab(self.projectName,name))
end
self.tabs = tabs
end
function Tabs:at(index)
return self.tabs[index]
end
Tab = class()
function Tab:init(project,name)
self.project = project
self.name = name
end
function Tab:classes()
if not self.classtable then
self.classTable = self:getClassTable()
end
return self.classTable
end
function Tab:getClassesInTab(tabName)
local tabContents = readProjectTab(self.project..":"..tabName)
local pattern = PatternDictionary():classDefinition()
local matchResult = tabContents:gmatch(pattern)
local classes = {}
for name in matchResult do
table.insert(classes,name)
end
return classes
end
function Tab:getClassTable()
local classes = {}
local tabs = self:getTabs()
for _i,tab in ipairs(tabs) do
local tabClasses = self:getClassesInTab(tab)
for _i, name in ipairs(tabClasses) do
table.insert(classes,name)
end
end
return classes
end
function Tab:getTabs()
return listProjectTabs(self.project)
end
function Tab:numberOfClasses()
return #self:classes()
end
function Tab:tabName()
return self.name
end
So Tab has a member variable classTable that it lazy inits with getClassTable(). We’ll follow that pattern. I am becoming less enamored of the lazy init. It seems like a premature optimization.
This observation is underlined in my mind when I notice that the reference to the classtable in classes() should be to classTable, so that the lazy init inits every time you call classes. Meh.
But we’re here on another mission. Refactoring comes soon.
I’d best write the test first.
_:test("Methods", function()
local tabs = Tabs("SampleStatsProject")
local tab = tabs:at(2)
local methods = tab:methods()
_:expect(#methods).is(1)
end)
The answer isn’t 1. It is probably three or four. We’ll check back in the sample project in a moment.
The test asks for methods
3: Methods -- TabTests:33: attempt to call a nil value (method 'methods')
Expletive it. I’m not going to lazy init this one. That will give me a pattern for undoing the other lazy init.
function methods()
return self.methodTable
end
Of course I still have to get them.
function Tab:init(project,name)
self.project = project
self.name = name
self.classTable = nil
self.methodTable = self:getMethods()
end
I inited classTable to nil. There’s no test calling for that. If I were a fanatic about test-driving, I’d have one. I’m not. I am human, as far as you know. I skate sometimes on thin ice. Is that wise? Almost certainly not.
I decide to name the method to match the other one:
self.methodTable = self:getMethodTable()
The more important thing is to write it:
function Tab:getMethodTable()
local methods = {}
local tabs = self:getTabs()
for _i,tab in ipairs(tabs) do
local tabMethods = self:getMethodsInTab(tab)
for _i, name in ipairs(tabMethods) do
table.insert(methods,name)
end
end
return methods
end
I’m doing the next-worst thing to copy-paste here, copying the code manually. I think these classes and methods aren’t what we really want the stats maker to do, but we’ll work on that when this works, because we’ll then have enough stuff computed to begin to think about organizing it.
function Tab:getMethodsInTab(tabName)
local tabContents = readProjectTab(self.project..":"..tabName)
local pattern = PatternDictionary:methodDefinition()
local matchResult = tabContents:gmatch(pattern)
local methods = {}
for start,stop,className,methodName,args in matchResult do
local method = MethodDefinition(className,methodName,args)
table.insert(methods, method)
end
return methods
end
I posit a new class, MethodDefinition. I’m writing rather more code than one expects for a microtest, but then again, this isn’t a microtest I’m writing. I feel that I know what I’m doing. Can run the test now, expecting to fail on MethodDefinition, but maybe the test will find something else first.
We get lots of fails on MethodDefinition, because I’m not lazy initing the thing. That does mean that all the tests will fail until this one works at least well enough to return something.
MethodDefinition = class()
function MethodDefinition:init(className, methodName, args)
self.className = className
self.methodName = methodName
self.args = args
end
That should get the show on the road. Yes. Error is:
3: Methods -- TabTests:33: attempt to call a nil value (method 'methods')
Easily done:
function Tab:methods()
return self.methodTable
end
I had forgotten to put the Tab: on the front of that one. Bril. That’s why I make the zero bucks writing these things.
Test.
3: Methods -- Actual: 5, Expected: 1
That’s a lot like the error I was expecting. Now to check the answer by looking at the sample. How fortunate: there are five of them in there. But we can do better.
I think we shouldn’t be returning raw collections of e.g. MethodDefinition objects, but some kind of smart object that can give us some help. I think I’d like to work my way up to that, however.
Belay that. I was thinking I could use my nice table-searching functions inside my smart object, but those functions aren’t included in this project, and they aren’t currently broken out for use as a dependency. Let’s drive the smart collections top down.
I think that at some point, I’ll want a ClassDefinition object that has method definitions in it. But we’re working on methods now, so let’s drive out a MethodsCollection.
_:test("Methods", function()
local tabs = Tabs("SampleStatsProject")
local tab = tabs:at(2)
local methodCollection = tab:methods()
_:expect(methodCollection:count()).is(5)
end)
I changed the test to expect a methods collection, and for that object to understand count(). Test drive:
3: Methods -- TabTests:34: attempt to call a nil value (method 'count')
We need to change our code to create a MethodCollection, and to store it in the Tab. So …
function Tab:init(project,name)
self.project = project
self.name = name
self.classTable = nil
self.methodCollection = self:getMethodCollection()
end
Now I need to implement getMethodCollection(). I had in mind reworking getMethodTable but hell, let’s just use it.
function Tab:getMethodCollection
return MethodCollection(self:getMethodTable)
end
And of course …
MethodCollection = class()
function MethodCollection:init(methodTable)
self.methodDefinitions = methodTable
end
function MethodCollection:count()
return #self.methodDefinitions
end
I am optimistic that the test will pass. That’s better than hopeful, worse than sure. Yes, well, two errors in a three line method. I got the end right.
function Tab:getMethodCollection()
return MethodCollection(self:getMethodTable())
end
Now I get an error:
3: Methods -- TabTests:34: attempt to index a nil value (local 'methodCollection')
That’s here:
_:test("Methods", function()
local tabs = Tabs("SampleStatsProject")
local tab = tabs:at(2)
local methodCollection = tab:methods()
_:expect(methodCollection:count()).is(5)
end)
What is Tab:methods?
function Tab:methods()
return self.methodTable
end
Perhaps we should use the correct member variable.
function Tab:methods()
return self.methodCollection
end
Test. Runs. We have the right count. Should we commit, or extend the test? We should commit: Initial MethodCollection.
Now lets extend the test, essentially driving out some useful methods in the MethodCollection class.
_:test("Methods", function()
local tabs = Tabs("SampleStatsProject")
local tab = tabs:at(2)
local methodCollection = tab:methods()
_:expect(methodCollection:count()).is(5)
local methodsInSampleCode = methodCollection:methodsForClass("SampleCode")
_:expect(methodsInSampleCode:count()).is(4)
end)
I’m positing a new method on MethodCollection, methodsForClass. Is that name redundant? Let’s think about that. Later.
3: Methods -- TabTests:35: attempt to call a nil value (method 'methodsForClass')
Write it. I have in mind that it produces another MethodCollection, not a raw table. Might be useful.
function MethodCollection:methodsForClass(className)
local result = {}
for _i,meth in ipairs(self.methodDefinitions) do
if meth.className == className then
table.insert(result, meth)
end
end
return MethodCollection(result)
end
I expect this to work. Yes, I am that confident.
3: Methods -- Actual: 0, Expected: 4
This is why we test. I have no idea why this didn’t work. I’ll resort to printing some info.
The prints tell me that the MethodDefinition objects are defined with the args in className. I suspect I included start and stop in there and the gmatch method doesn’t return those.
function Tab:getMethodsInTab(tabName)
local tabContents = readProjectTab(self.project..":"..tabName)
local pattern = PatternDictionary:methodDefinition()
local matchResult = tabContents:gmatch(pattern)
local methods = {}
for start,stop,className,methodName,args in matchResult do
local method = MethodDefinition(className,methodName,args)
table.insert(methods, method)
end
return methods
end
Sure enough. Fix:
function Tab:getMethodsInTab(tabName)
local tabContents = readProjectTab(self.project..":"..tabName)
local pattern = PatternDictionary:methodDefinition()
local matchResult = tabContents:gmatch(pattern)
local methods = {}
for className,methodName,args in matchResult do
local method = MethodDefinition(className,methodName,args)
table.insert(methods, method)
end
return methods
end
Now can I get an OK? Yes. Remove all those prints.
Let’s commit this, but then we need to talk. Commit: MethodCollection:methodsForClass()
Retro
The test quickly discovered that the MethodDefinitions weren’t correct. But not quickly enough. This morning I have repeatedly built code that was not implied by the tests, and often, though not always, that code wasn’t quite correct, only to be discovered later.
And even if “often” is overstated, it happened at least once, and that is why, when I’m on my best game, I stick very closely to the TDD “rule”:
Write only the code necessary to make the test pass.
When we write code that isn’t necessary to the test passing, that code is untested. That means that when it contains a mistake, quite often no test discovers the problem. Now, with luck, a later test will discover that code to be broken, but even when that happens, the error is now further from the test we’re thinking about, and the cause is harder to find.
My habits with TDD are not ideal. I often write code that’s not implied by the test I’m writing, and more often than I can justify, that uncovered code is wrong. In one case, a defect stayed in the Dung system from May until October before I found it.
TDD feels slow, but finding a bug in October that was implemented in May, friends, that’s slow!
Anyway, it’s Saturday, and I am trying to take it easy. We’ll sum up and close out now.
Summary
Our overall mission is to write a handy little Codea program to collect and display “statistical” information about Codea projects. Number of classes, etc etc. We have really only a couple of information packets in place. We can find all the classes, and we can find all the methods. For the methods, we have a MethodDefinition object that includes the class of the method, its name, and a raw string of the arguments to the method.
And we have a MethodCollection object that can tell us how many elements there are in itself, and can select method definitions by class name.
Supporting those we have a Tabs class containing a collection of Tab instances, a Tab class whose instances include a raw collection of class name strings, and a MethodCollection, of the classes and methods found on the given tab.
We’ll want to have the raw class name collection become a ClassCollection, and for it to contain instances of ClassDefinition, which will, in the fullness of time, include all the MethodDefinitions belonging to that class.
Somewhere along there, we’ll need to do something to provide structure outside the Tab, such as a collection of classes found in the whole program. And we’ll want all the methods for a given class to be found in that outside collection.
I’m not sure what that implies. Probably that there is a master ClassCollection and that we put the classes into that, and that we find the appropriate class and put our MethodCollection in there.
We may break the connection to the Tab notion, such that while we will certainly process tab by tab one way or another, we will probably not materialize that collection “high up” in the program.
All that remains to be discovered. I’m just speculating, as one does. For now, things are moving nicely. I think we’ll look “higher up” next time.
See you then!