Dungeon 186 - An addition to the Making App.
Dave1707 from the Codea forum provided an interesting program. I would like to bring it under control.
In refactoring and generally trying to improve the Dung program, I often wish for some analytical features from Codea, such as information on the number of classes, methods, tests, and so on. The more we know about those things, the better sense we can get about what the program wants us to do next.
So yesterday I asked whether there is a way to get the number of classes in a Codea program. Naturally, I figured that if there was, it would work by counting the classes and after that we’d have a handle on getting more and more information. Sort of a Pandora’s box question.
Dave1707, who is always helpful and often surprising, came right back and pasted in a program that we’ll be looking at this morning, and perhaps off and on in the future. Long story short, the program can be pointed at any Codea project and it produces an on-screen display like this:
This is really nifty in my view. I like it. I would like to include it in my development cycle, even though I’m not sure just how and when it will come in handy. Certainly knowing the protocol for all my classes has value.
There are things that could be better. As written, the program displays things in the order of the program’s tabs, and inside a tab, displays classes, functions, and methods in the order they appear. That’s not bad, but it’s not always what we want.
For a general report, we’d probably like to see things in alphabetical order by class name, and method name within class. We’d probably like to have some elementary statistical information, like the number of methods and number of lines in the class (and method). Those may not be easy to do, starting from here.
We’ll look at the program in a moment, but suffice to say that this information is produced on a tab by tab line by line basis. Aside from a bit of special handling for comments, the program just detects lines containing “class” or “function” and displays them, tabbed nicely.
The program never really knows that it is processing a given class, never really knows that a method is part of a class, and has absolutely no notion of where the end of a function is. There’s no real parsing of the program.
And, in my view, that is as it should be. Now if Codea had calls you could make to inspect the program, returning patches of code or a parse tree or something, I might think differently. But what we have here is a simple program that just spins through the lines of your program, emitting interesting lines in order.
Wild, uncontrolled thought
My plan going in this morning was to refactor Dave’s program, and to get it under test. But now, I find myself thinking “It is simple. Maybe it would be better to write a program of my own, copyingXXX adopting Dave’s ideas.
I always prefer to program something new: this is one of my good, and bad, properties. But if I can successfully get this program refactored and under test, we may have a nice small lesson in how that can be done.
OK, we’ll stick with Plan A.
Dave’s Program
Here, in its entirety, is Dave’s program:
viewer.mode=FULLSCREEN
function setup()
project="D2"
lst=listProjectTabs(project)
for a,b in pairs(lst) do
str=readProjectTab(project..":"..b)
end
font("Courier")
textMode(CORNER)
dy=0
tab={}
r=listProjectTabs(project)
for a,b in pairs(r) do
str=readProjectTab(project..":"..b)
table.insert(tab,"\n")
table.insert(tab,"\n")
table.insert(tab,project.." : "..b)
b1=1
s=1
comment=false
startComment=false
while true do
s,e=string.find(str,"\n",s)
if s==nil then
break
end
s=e+1
str1=string.sub(str,b1,e)
sc1,ec1=string.find(str1,"%-%-%[%[")
if sc1 ~= nil then
startComment=true
end
--sc2,ec21=string.find(str1,"%-%-%]%]")
sc2,ec21=string.find(str1,"%]%]")
if sc2 ~= nil then
startComment=false
end
comment=false
sc3,ec3=string.find(str1,"%-%-")
if sc3 ~= nil then
comment=true
sc4,ec4=string.find(str1,"class")
if sc4~=nil and sc4<sc3 then
comment=false
end
sc5,ec5=string.find(str1,"function")
if sc5~=nil and sc5<sc3 then
comment=false
end
end
process=true
if comment or startComment then
process=false
end
if process then
s1,e1=string.find(str1,"class")
if s1~=nil then
s2,e2=string.find(str1,")")
if s2~= nil then
str1=removeSpaces(string.sub(str1,1,e2))
table.insert(tab," "..str1)
b1=e2+1
end
else
s2,e2=string.find(str1,"function ")
if s2~=nil then
s3,e3=string.find(str1,")")
if s3~= nil then
table.insert(tab," "..string.sub(str1,s2,e3))
b1=e3+1
end
end
end
if s==nil then
break
end
end
if e~=nil then
b1=e+1
end
end
end
table.insert(tab,"\n")
table.insert(tab,"\n")
table.insert(tab,"END of "..project)
end
function removeSpaces(s)
for z=1,#s do
if string.sub(s,z,z)~=" " then
return(string.sub(s,z))
end
end
end
function draw()
background(40, 40, 50)
fill(255)
for a,b in pairs(tab) do
text(b,100,HEIGHT-a*20+dy)
end
end
function touched(t)
if t.state==BEGAN or t.state==MOVING then
dy=dy+t.deltaY
if dy<0 then
dy=0
end
end
end
What do we see here? Well, we see that the setup function is just shy of 90 lines long, including a for loop of about 73 lines, containing a while loop of about 62 lines. Inside there, we have a 22-line if nest, containing an if-else, each branch another one or two levels deep in if.
Prior to that there are 19ish lines deciding whether we’re currently processing one or more comment lines (and therefore shouldn’t look for “class” or “function”. To make that decision, those lines also look for “class” and “function”, in case we have a class or function line containing a comment but not commented-out.
The program uses a few key state variables, notably startComment and comment, the former having to do with multi-line comments “–[[ –]]” and the latter with single line comments “–”.
Wait up here, Ron. We see that?
OK, you got me. We don’t just oh yes here we go this is easy “see” it. I looked hard at the program last night, and in the light of that, again this morning. It’s not bad code at all, it’s very straightforward for the most part, and it’s pretty easy to understand.
I think I could even change it, at least to handle some of the things I might like. I’d have to be careful, but hey, you’re supposed to be careful when you’re programming.
Important Note
I want to say clearly here, that this is a very nice little program, and what I’m saying, and what I’m about to do, are in no way a criticism of Dave or his program. What I’m going to do is to use a very reasonable procedural program to demonstrate how we might get a much more complicated and messy program under test, and under team control.
Often we just use the tools we’ve found “out there” somewhere. I think that usually those tools can be made more useful when we can get them under our own team’s control and modify them to better fit our local needs.
That’s what I’m going to do here, and I’m fortunate to have a habitable, working program to start with.
Thanks, Dave!
Where We Might Be Going
I propose to get this program running with associated unit tests, and to move it to a more object-oriented style, ultimately, I hope, turning it into a small set of classes that can be included into any program as a dependency and used to display useful info about that program.
I have some basic ideas about how to do that. Here are a few:
- Put the app into Working Copy so that I can version it;
- Include CodeaUnit as a dependency, so that I can write tests for it;
- Move the existing program into a separate function, and perhaps into classes;
- Use Codea’s project data or local data capability to store golden master versions of the output, for large-scale testing.
I’m sure that more ideas will arise.
I think this may take me a couple of passes. Right now, I have the program in a project called Making, which is such a good name that I am tempted to name this something else. It’s in the form you saw above, all that code in one Main tab.
My first move is going to be to move the guts into a separate tab from Main, as a function that Main calls. That should be “easy”.
And it was.
function setup()
report()
end
That’s the Main setup, and the rest of the code is in a new tab, Report, containing the whole program, as written, inside
function report()
...
end
I’ll move over the removeSpaces function as well.
Next, I think I’ll add CodeaUnit as a dependency on this project. Doing that just means pressing Codea’s “DO” button, selecting “Dependencies”, and checking the box on CodeaUnit, once you find it in the list.
Now I should find that the program has a CodeaUnit button that will run the tests we don’t have.
Let’s put this baby into Working Copy now, then start working.
I close Codea for this step, because file access is still pretty weird on iPads, and this ensures that the code is all saved where Working Copy can find it.
In Working Copy’s Repo list, hit + to add one, Init new project Making. In Repo list, long-press Making, and set up project sync.
It gives me a weird diagnostic that I’ve not seen before about Missing directory, but I’m going to ignore that. I think it’s warning me that I’m looking into Codea’s folders, not Working Copy’s. I’ll ask Anders later.
The commit comes up, looking right. Commit: initial commit.
Let’s add a test tab and a hookup test.
-- RJ 20210525
-- Tests for Making App
function testMakingApp()
CodeaUnit.detailed = true
_:describe("Making App", function()
_:before(function()
end)
_:after(function()
end)
_:test("HOOKUP", function()
_:expect("Foo", "testing fooness").is("Bar")
end)
_:test("Floating point epsilon", function()
_:expect(1.45).is(1.5, 0.1)
end)
end)
end
The tests run, providing one failure (Foo~=Bar) and one success. We’ll fix that up. I generally remove the floating point one, which is just there to remind me where the epsilon goes in an FP test. And I change “Bar” to “Foo” to make that test pass.
Tests are green. (Whee!)
Now I think we’ll start to do some actual work. It seems clear to me that the bulk of the work is done on a per-tab basis, and we should be able to test just a single tab for most of our testing. Let’s look at the function we have in hand:
function report()
project="D2"
lst=listProjectTabs(project)
for a,b in pairs(lst) do
str=readProjectTab(project..":"..b)
end
font("Courier")
textMode(CORNER)
dy=0
tab={}
r=listProjectTabs(project)
for a,b in pairs(r) do
str=readProjectTab(project..":"..b)
table.insert(tab,"\n")
table.insert(tab,"\n")
table.insert(tab,project.." : "..b)
b1=1
s=1
comment=false
startComment=false
while true do
s,e=string.find(str,"\n",s)
if s==nil then
break
end
s=e+1
str1=string.sub(str,b1,e)
sc1,ec1=string.find(str1,"%-%-%[%[")
if sc1 ~= nil then
startComment=true
end
--sc2,ec21=string.find(str1,"%-%-%]%]")
sc2,ec21=string.find(str1,"%]%]")
if sc2 ~= nil then
startComment=false
end
comment=false
sc3,ec3=string.find(str1,"%-%-")
if sc3 ~= nil then
comment=true
sc4,ec4=string.find(str1,"class")
if sc4~=nil and sc4<sc3 then
comment=false
end
sc5,ec5=string.find(str1,"function")
if sc5~=nil and sc5<sc3 then
comment=false
end
end
process=true
if comment or startComment then
process=false
end
if process then
s1,e1=string.find(str1,"class")
if s1~=nil then
s2,e2=string.find(str1,")")
if s2~= nil then
str1=removeSpaces(string.sub(str1,1,e2))
table.insert(tab," "..str1)
b1=e2+1
end
else
s2,e2=string.find(str1,"function ")
if s2~=nil then
s3,e3=string.find(str1,")")
if s3~= nil then
table.insert(tab," "..string.sub(str1,s2,e3))
b1=e3+1
end
end
end
if s==nil then
break
end
end
if e~=nil then
b1=e+1
end
end
end
table.insert(tab,"\n")
table.insert(tab,"\n")
table.insert(tab,"END of "..project)
end
The first thing I notice is that loop at the beginning:
lst=listProjectTabs(project)
for a,b in pairs(lst) do
str=readProjectTab(project..":"..b)
end
What’s that about? It seems to have no purpose. I remove it. (I also drop Dave a note asking whether it has a purpose that I don’t recognize.)
Now I want to begin to test a single tab, so with the existing display of D2 showing, I extract the inside of the report program into a new function:
function report()
project="D2"
font("Courier")
textMode(CORNER)
dy=0
tab={}
r=listProjectTabs(project)
for a,b in pairs(r) do
str=readProjectTab(project..":"..b)
processTab(str)
end
table.insert(tab,"\n")
table.insert(tab,"\n")
table.insert(tab,"END of "..project)
end
function processTab(str)
table.insert(tab,"\n")
table.insert(tab,"\n")
table.insert(tab,project.." : "..b)
b1=1
s=1
comment=false
startComment=false
while true do
s,e=string.find(str,"\n",s)
if s==nil then
break
end
s=e+1
str1=string.sub(str,b1,e)
sc1,ec1=string.find(str1,"%-%-%[%[")
if sc1 ~= nil then
startComment=true
end
--sc2,ec21=string.find(str1,"%-%-%]%]")
sc2,ec21=string.find(str1,"%]%]")
if sc2 ~= nil then
startComment=false
end
comment=false
sc3,ec3=string.find(str1,"%-%-")
if sc3 ~= nil then
comment=true
sc4,ec4=string.find(str1,"class")
if sc4~=nil and sc4<sc3 then
comment=false
end
sc5,ec5=string.find(str1,"function")
if sc5~=nil and sc5<sc3 then
comment=false
end
end
process=true
if comment or startComment then
process=false
end
if process then
s1,e1=string.find(str1,"class")
if s1~=nil then
s2,e2=string.find(str1,")")
if s2~= nil then
str1=removeSpaces(string.sub(str1,1,e2))
table.insert(tab," "..str1)
b1=e2+1
end
else
s2,e2=string.find(str1,"function ")
if s2~=nil then
s3,e3=string.find(str1,")")
if s3~= nil then
table.insert(tab," "..string.sub(str1,s2,e3))
b1=e3+1
end
end
end
if s==nil then
break
end
end
if e~=nil then
b1=e+1
end
end
end
That is surprisingly tricky to do, because in Codea, I can’t see both ends of the function at the same time in the editor.
I quickly learn that I need to pass “b” into the function:
function report()
project="D2"
font("Courier")
textMode(CORNER)
dy=0
tab={}
r=listProjectTabs(project)
for a,b in pairs(r) do
str=readProjectTab(project..":"..b)
processTab(str,b)
end
table.insert(tab,"\n")
table.insert(tab,"\n")
table.insert(tab,"END of "..project)
end
function processTab(str,b)
table.insert(tab,"\n")
table.insert(tab,"\n")
table.insert(tab,project.." : "..b)
...
The program is working again.
We should commit: Tests tab added, reportTab function broken out.
Now we need to create something we can actually test. I propose to have a new tab in this project. But I do have a concern.
I imagine using this project by including it as a dependency in the project we’re actually working on, and calling it with a button or something like that. For that to be ideal, we don’t want a lot of extra tabs in our app.
I think we’ll burn that bridge when we come to it. Right now, let’s get this thing under control.
I start with this new tab, named “Sample”:
-- Sample
-- RJ 20250525
function sample(a,b,c)
local x = 15
end
Barker = class()
function Barker:init(dog, name)
self.dog = dog
self.name = name
end
function Barker:speak()
SoundMachine:sound(self:woof())
end
function Barker:woof()
return "woof"
end
Now let’s stop running the program from setup, and start running it from the tests.
It takes me a little messing about, since our draw and touch functions in Main are relying on some globals that the original sets up. I wind up here:
_:test("Sample", function()
project = "Making"
font("Courier")
textMode(CORNER)
dy=0
tab={}
local str = readProjectTab("Sample")
processTab(str,"Sample")
end)
The general scheme here is that the original program computes the table tab and the draw displays it, and touch, using dy, scrolls it. We don’t exactly want that, though we certainly do want to view the output.
Irritatingly, the output comes out under the pull-over part of Codea:
I think I’ll just see where the tabbing is and tab it further out. From looking at that picture, I think tabbing in an additional number of spaces slightly longer than “Making : Sample” should do it. I’ll try 20 spaces. How does the tabbing work? It’s of course quite ad-hoc. I’ll do this:
function processTab(str,b)
local extra = " "
table.insert(tab,"\n")
table.insert(tab,"\n")
table.insert(tab,extra..project.." : "..b)
...
table.insert(tab,extra.." "..str1)
...
table.insert(tab,extra.." "..string.sub(str1,s2,e3))
...
Those seem to be the only lines that insert actual text, and I just prepended the new extra spaces. Let’s see how it looks.
That’ll do. Let’s add some stuff to sample, including some comments that we think the program can handle:
-- Sample
-- RJ 20250525
--[[ some kind of comment
more comment
--]]
function sample(a,b,c)
local x = 15
end
Barker = class()
function Barker:init(dog, name)
self.dog = dog
self.name = name
end
function Barker:speak()
SoundMachine:sound(self:woof())
end
function Barker:woof()
return "woof"
end
Porker = class()
function Porker:init(piggie, piggieName)
end
--[[
function shouldIgnore()
end
--]]
math.nothing = function(value)
return value
end
function nothing(value)
return value
end
If everything were perfect, the shouldIgnore function would not show up, and the math.nothing one would show up. In fact the latter does not show up. If we want it, we’re going to have to do some much more sophisticated parsing.
There’s another kind of function to think about, the anonymous kind, such as we get in a test:
_:test("HOOKUP", function()
_:expect("Foo", "testing fooness").is("Foo")
end)
This is really the same as the math.nothing one. The function has no name (like the girl who has no name). As such, we can decide not to display it, and given that we’re mostly interested in classes and methods, I think we’re fine to do that.
Here’s the output from things as they stand now:
So that’s good. Let’s turn our current output into a golden master, and then check it.
I think I’ll add a new button. What should it do? Well, it should make and save a golden master. Unfortunately, Codea can only store a string or a number as local, project, or global data. We’ll need to compute a string to use for our comparison.
Let’s first write a check that compares our output tab to the master. This is going to make me want to change how my outer calling sequence works here.
Here’s my new processTab:
function processTab(str,b)
local tab = {}
local extra = " "
table.insert(tab,"\n")
table.insert(tab,"\n")
table.insert(tab,extra..project.." : "..b)
b1=1
...
if e~=nil then
b1=e+1
end
end
return tab
end
I put that into the report function thusly:
function report()
project="D2"
font("Courier")
textMode(CORNER)
dy=0
tab={}
r=listProjectTabs(project)
local tabData
for a,b in pairs(r) do
str=readProjectTab(project..":"..b)
tabData = processTab(str,b)
for i,line in ipairs(tabData) do
table.insert(tab,line)
end
end
table.insert(tab,"\n")
table.insert(tab,"\n")
table.insert(tab,"END of "..project)
end
I “just” return the result of processTab into a local table tabData, and append it to the output table tab. I added a button to run the full report on “D2” and it works.
We are still refactoring without much support from tests, so we’re not where we’d like to be, yet. One thing at a time: I’m working toward the golden master. I have this:
function setup()
tab = {}
dy = 0
--report()
parameter.action("Make Golden", function()
makeGolden()
end)
parameter.action("report D2", function()
report()
end)
end
function makeGolden()
if #tab > 0 then
saveLocalData("tab", table.concat(tab))
end
end
Conveniently, table.concat concatenates the print values of a table into a string. We can actually test this now, but we aren’t going to like it much.
_:test("Sample", function()
project = "Making"
font("Courier")
textMode(CORNER)
dy=0
tab={}
local str = readProjectTab("Sample")
local result = processTab(str,"Sample")
tab = result
local tabString = table.concat(result)
local tstString = readLocalData("tab")
_:expect(tabString).is(tstString)
end)
First time thru, this should fail, since the read won’t return anything good. After that, if I like the output, I’ll press the golden master button and save that copy. Oh. Except it won’t be right, I’m not saving the right thing. Let me try my makeGolden again:
function makeGolden()
local str = readProjectTab("Sample")
local result = processTab(str,"Sample")
local resultString = table.concat(result)
saveLocalData("tab", resultString)
end
That’s more like it. Let’s test and see what we see.
Test fails first time through, as expected:
2: Sample -- Actual:
Making : Sample function sample(a,b,c) Barker = class() function Barker:init(dog, name) function Barker:speak() function Barker:woof() Porker = class() function Porker:init(piggie, piggieName) function nothing(value), Expected: nil
We seem to have stripped out all the newlines. Let’s presume that won’t harm us.
Now to press the Make Golden button, then run again.
It doesn’t say anything, and probably should, or flash the screen or something. Anyway, it probably saved, so let’s run the tests again:
2: Sample -- OK
Test passes! We have a Golden Master compared with our output.
Ah. Reading the FM, I see that table.concat can take a separator. I’m all for that. Let’s do it.
_:test("Sample", function()
project = "Making"
font("Courier")
textMode(CORNER)
dy=0
tab={}
local str = readProjectTab("Sample")
local result = processTab(str,"Sample")
tab = result
local tabString = table.concat(result, "\n")
local tstString = readLocalData("tab")
_:expect(tabString).is(tstString)
end)
end)
end
function makeGolden()
local str = readProjectTab("Sample")
local result = processTab(str,"Sample")
local resultString = table.concat(result, "\n")
saveLocalData("tab", resultString)
end
Do again:
2: Sample -- Actual:
Making : Sample
function sample(a,b,c)
Barker = class()
function Barker:init(dog, name)
function Barker:speak()
function Barker:woof()
Porker = class()
function Porker:init(piggie, piggieName)
function nothing(value), Expected:
Making : Sample function sample(a,b,c) Barker = class() function Barker:init(dog, name) function Barker:speak() function Barker:woof() Porker = class() function Porker:init(piggie, piggieName) function nothing(value)
Nicer. Make the Master. Test again, expecting OK.
2: Sample -- OK
OK, let’s commit: test uses Sample tab. Can make and check Golden Master from local data.
What Now?
I can think of three things to do now.
- Stop, because between interruptions and such, our morning session has run long enough;
- Start working on the necessary changes to make this program useful in making D2;
- Refactor a bit for clarity.
From the viewpoint of personal benefit, I’d like to stop soon. From the viewpoint of project benefit, the sooner I bring this new find back to the team, the sooner it can be useful.
So let’s at least think about item #2, making it useful for D2.
If I were just to change the Main thus:
function setup()
tab = {}
dy = 0
--report()
parameter.action("Make Golden", function()
makeGolden()
end)
parameter.action("report D2", function()
report()
end)
end
When we run this app and press the “report D2” button, we get the original output we saw at the top of this article, the full report for the D2 program. Just for fun, I’ll display it in multiple panes (or pains) below. It takes 19 photos to take it all in: I’ll just show a few. We can see that this tool will need enhancement, but it’s fun and enlightening just to scroll through and observe things.
I think we’ll declare the program somewhat useful as it is, and sum up:
Working Effectively with Legacy Code
That title is the title of one of my most admired books, by Michael Feathers. It is a large compendium of techniques for getting legacy code under test and refactoring it into better shape.
The book includes examples in C++, Java, and C#, and its ideas are more generally applicable. Here in this article, we’ve just begun to scratch the surface of the good material you’ll find in WELC, should you decide to get a copy, which I recommend to anyone dealing with messy legacy code.
Here, we’ve done just a few things. We pulled out a function that processes a single string of code, typically a Codea tab’s worth. As we go forward with the program, we’ll surely begin passing in smaller snippets of code.
We saved a local “golden” copy of the test’s output, and compare its current output to that, emitting an error if they don’t match. If our tests are small enough, that may suffice. If not, we may find that we want to do some kind of “diff” or other useful display of what has happened.
Frankly, I’d rather not enhance the error-reporting, instead using small enough tests to make reading the output easy. We’ll see how that goes.
Looking forward, I suspect we’ll want the thing to generate some summary information, tab, class, number of methods, lines of code, and then to allow us to drill down to the details. I expect that we’ll be able to do a decent job of that, although not a perfect one.
We’ll probably assume that we’re presented with a well-formatted file, so that you can expect a correct line count if you give us this:
function foo(a)
local b = a*a
if b < a then
return "small"
else
return "large"
end
end
But not if you give us this:
function foo(a)
local b = a*a
if b < a then return "small"
else return "large" end end
We’ll see. I don’t think we want to write, nor steal, a lua syntax analyzer.
Anyway, we have a nice addition to our toolset, big thanks to Dave, and it’s starting to get into condition to be refactored and extended, small thanks to me.
See you next time. Here’s today’s code, and a few of the 19 pictures:
