Dungeon 189
One more challenge in the class/method finder, then maybe we’ll think about what we actually need in our Making App.
Dave pointed out that Lua multi-line comments are actually more complex than just “–[[” commenting until “]]”. They can be nested. Of course. Everyone wants nested comments. It’s done with multiple, matching, equal signs between the opening and closing braces. So “–[==[ blah blah ]==]”.
Our original program didn’t deal with them. Dave doesn’t use them, and not only do I not use them, I probably abhor them. Still, it’s an interesting challenge and one that I think we can readily meet.
I just love an interesting challenge that can readily be met, and if it turns out to be difficult, I’ll tell Tech Support to say “don’t do that” when users complain about the lack of support for nested comments.
The approximate plan is to put nested comments into the Sample, expecting nothing to come out where they are, and if something does come out, which it will at first, change the pattern to deal with the nesting.
Here goes. I added this to Sample:
--[==[ very nested comments
triple_nest_broke_top = 0
--[=[ nested comments
double_broke_top = 0
-[[ plain old multi-line
plain_broke = 0
--]]
double_broke_bot = 0
--]=]
triple_next_broke_bot = 0
--]==]
This should comment out all that stuff. We’ll test and expect an error. Test runs. I am surprised. But sometimes Codea doesn’t save some tabs. Try Save & Run.
Hm, Test still passes. I am surprised. Did I already make this work?
No:
function removeComments(prog)
local step1 = prog:gsub("%-%-%[%[.-%]%]","") -- must be first
local step2 = step1:gsub("%-%-.-\n","\n")
return step2
end
That should match two brackets preceded by two dashes, anything (shortest), two brackets. It should not do anything with “[=[”.
OK, next step, I’ll print the sample after comment removal.
Ah, my bad. We won’t get an error unless we produce some output inside the comments. Change the Sample:
--[==[ very nested comments
triple_nest_broke_top = class()
--[=[ nested comments
double_broke_top = class()
-[[ plain old multi-line
plain_broke = class()
--]]
double_broke_bot = class()
--]=]
triple_next_broke_bot = class()
--]==]
That should fail the test. That’s more like it:
1: Sample -- Actual:
Making : Sample
function sample(a,b,c)
triple_nest_broke_top = class()
double_broke_top = class()
plain_broke = class()
double_broke_bot = class()
triple_next_broke_bot = class()
Barker = class()
function Barker:init(dog, name)
function Barker:speak()
function Barker:woof()
Porker = class(Animal)
function Porker:init(piggie, piggieName)
function nothing(value)
function bizarreCommentNotFound(), Expected:
Making : Sample
function sample(a,b,c)
Barker = class()
function Barker:init(dog, name)
function Barker:speak()
function Barker:woof()
Porker = class(Animal)
function Porker:init(piggie, piggieName)
function nothing(value)
function bizarreCommentNotFound()
I am surprised by `plain_broke = class() coming out. Ah. A missing dash. Should be:
--[==[ very nested comments
triple_nest_broke_top = class()
--[=[ nested comments
double_broke_top = class()
--[[ plain old multi-line
plain_broke = class()
--]]
double_broke_bot = class()
--]=]
triple_next_broke_bot = class()
--]==]
Again:
1: Sample -- Actual:
Making : Sample
function sample(a,b,c)
triple_nest_broke_top = class()
double_broke_top = class()
double_broke_bot = class()
triple_next_broke_bot = class()
Barker = class()
function Barker:init(dog, name)
function Barker:speak()
function Barker:woof()
Porker = class(Animal)
function Porker:init(piggie, piggieName)
function nothing(value)
function bizarreCommentNotFound(), Expected:
Making : Sample
function sample(a,b,c)
Barker = class()
function Barker:init(dog, name)
function Barker:speak()
function Barker:woof()
Porker = class(Animal)
function Porker:init(piggie, piggieName)
function nothing(value)
function bizarreCommentNotFound()
Yes, that error looks like I’d expect it to look.
What’s our fix? Although Lua’s patterns are not full regex, they do support captures and capture matches. I think we can do this:
local step1 = prog:gsub("%-%-%[%(=-)[.-%]%1%]","")
I expect the tests to pass now. They do not. Perhaps I don’t know how to do the capture. I need a microtest. And to RTFM a bit more.
_:test("captures", function()
local s,e
local line = "xyz--[=[abc--[[def--]]ghi--]=]jkl"
local pattern = "%-%-%[(=-)%[.-%]%1%]"
s,e = string.find(line, "%-%-%[%[.-%]%]")
_:expect(s).is(12)
_:expect(e).is(22)
s,e = string.find(line,pattern)
_:expect(s).is(4)
_:expect(e).is(30)
end)
I think I mangled the pattern. The one in the test seems to work.
And that raises a general issue that I may have mentioned yesterday.
I think it’s a good practice to write microtests for pattern matches. They’re easier to get right, working in the small, and the test documents what’s supposed to happen. But when I finally get the pattern right … I have to copy/paste it back into the code. This is not a good thing. I could test this one here until it works but the real code isn’t any better.
I have an idea for this, but right now I’m sitting on an implementation. Let’s make it work, then make it right.
Copy the working pattern to the code:
function removeComments(prog)
local step1 = prog:gsub("%-%-%[(=-)%[.-%]%1%]","") -- must be first
local step2 = step1:gsub("%-%-.-\n","\n")
return step2
end
Run the tests, expecting a pass.
1: Sample -- OK
...
8 Passed, 0 Ignored, 0 Failed
OK, good to go. Commit: implemented nested multi-line comments.
Now let’s make the tests use the same patterns as the actual code. One way, and the way I think I’ll use, is to put the patterns into separate functions over on the Report tab, and use them in the tests. Another, slightly more efficient, would be to make them local to the Report, and provide functions to return them. Most efficient, and least advised from these quarters, would be to make the patterns global.
We only call these functions once per tab in the program being analyzed. I’ll go with the functions as they’ll be a bit less code.
function multiCommentPattern()
return "%-%-%[(=-)%[.-%]%1%]"
end
function removeComments(prog)
local step1 = prog:gsub(multiCommentPattern(),"") -- must be first
local step2 = step1:gsub("%-%-.-\n","\n")
return step2
end
_:test("captures", function()
local s,e
local line = "xyz--[=[abc--[[def--]]ghi--]=]jkl" s,e = string.find(line,multiCommentPattern())
_:expect(s).is(4)
_:expect(e).is(30)
end)
_:test("gsub removes --[[ comments", function()
local prog = "abc--[[def\nghi\njkl]]mno\npqr--[[stu\nvwx\nyza]]bcd\n"
local res,numberOfSubs = prog:gsub(multiCommentPattern(),"")
_:expect(res).is("abcmno\npqrbcd\n")
_:expect(numberOfSubs).is(2)
end)
Test. Works. Now the other one.
function singleCommentPattern()
return "%-%-.-\n"
end
function removeComments(prog)
local step1 = prog:gsub(multiCommentPattern(),"") -- must be first
local step2 = step1:gsub(singleCommentPattern(),"\n")
return step2
end
_:test("gsub removes -- comments", function()
local prog = "abcd--efgh ijkl\nMNOP--XXXX\nQRST\n"
local res,numberOfSubs = prog:gsub(singleCommentPattern(),"\n")
_:expect(numberOfSubs).is(2)
_:expect(res).is("abcd\nMNOP\nQRST\n")
end)
Test. Works. Commit: tests and code now use same patterns automatically.
So that’s nice. Quick Retro and then we’ll see if we want to do anything else.
Retrospective
If there’s a lesson here, it might be that matching patterns are tricky, difficult to get exactly right, and fail in odd ways. For me, that means that I will always do better if I at least test them with small patches of code, and better still if I create those patches of code as tests.
A smaller lesson, but one that I value, is the idea of creating functions or other public ways of accessing the functions, and using those to keep tests in sync with code.
I’m also glad to have learned a bit more about Lua’s patterns. I don’t have other advanced uses in mind, but it does seem somewhat clear that we could do some fairly fancy pattern recognition in our code, making it possible to build up some useful tools without too much investment.
Speaking of that, this is, what, the fourth day I’ve spent working on Dave’s little program, shaping it to my liking, with an eye to using it as part of my ongoing toolset. How many actual programming hours is that? Still less than a day. It takes me more time to type the article than it does the code, in most cases, unless I get stuck in some debugging pit.
I do think this has been a bit more refactoring than was strictly needed, but at the same time, I predict that we’ll find more aspects that need tidying up before we have a Making App component that we’re really satisfied with.
Anyway, good work today. Let’s at least plan a bit, if not actually do a bit.
Next Steps for Making App
Calling this a Making App is like calling four Lego bricks a house. Might work for a three-year old, but it’s not going to save the housing market. But it is a start.
Let’s look at things we’d like to be able to find out about an application.
- How many classes are there;
- How many methods are there;
- How many free functions are there;
- Where are the free functions; do they need better grouping;
- How many tests do we have;
- How many expects do we have;
- Given a class, what other classes does it instantiate;
- Given a method, what classes implement it;
- Given a partial method name, what methods match it;
- Given a method, what is its calling sequence;
- What globals does the app create;
- What variables are used but never assigned;
- What variables are assigned but never used;
- What is the distribution of method length;
- What is the distribution of number of methods by class;
Wow, there’s a product waiting to be built here. We could sell one to every Codea programmer in the world. We could make literally tens of dollars!
But there really are a lot of questions that it’s useful to know.
Some of these questions can be answered by a static analysis of the program, like our Making app does now, looking at the source code. Others, like which globals the app creates, would have to be done dynamically.
And even some of the static ones, we’d really like to be able to invoke from inside the Codea editor, on the program we’re working on. At present, we can’t have two Codea instances running at once, as far as I know.
It’ll take some time to sort out which of these ideas is most useful. Dave has written another useful little program, that produces a sources and uses report:
The left column is classes defined in each tab, the right column shows the lines instantiating classes. The program works by an exhaustive search of the source code looking for patterns including a known class name and “(something)”. Pretty straightforward. Takes a few seconds to run on the Dungeon program, so it’s fast enough.
I don’t think I’ll refactor this one. What I’ll do instead, I believe, is bring over the ideas, and probably some of the code, into this current Making app.
One thing the new program does that is quite nice is that it has a tiny but useful user interface. It lists projects down in the console, and you can type a project name into the text field, and then press search. A short delay, and there’s your report.
That’s just done, of course, with Codea’s nice parameter capability:
parameter.action("list projects",listProj)
parameter.text("project")
parameter.action("search",srch)
When you click the action buttons, those functions are called. The text button defines a global(!) named project, which is used in the function srch. Simple, but nice.
We may wish for a larger menu system as we go forward. Right now, no.
However, right now I do want one more thing. This one I’m going to build into the Dungeon program, because it needs to operate at run time. I’m going to find out what globals the program defines.
Let’s see what we can figure out.
Globals, What Are They?
We saw the other day that there are a lot of globals in Lua, 346 or something. So displaying them all won’t be helpful. Instead we’ll capture them at one point, then again at another, and difference the two collections, looking for new ones.
Shall I TDD this? Sure, why not. Should be easy.
New Tab: class GlobalAnalyzer
_:test("Detect New Globals", function()
local ga = GlobalAnalyzer()
ga:snapshot()
local new = ga:newGlobals()
_:expect(#new).is(0)
newGlobal = 5
new = ga:newGlobals()
_:expect(#new).is(1)
_:expect(new[1]).is("newGlobal")
newGlobal = nil
new = ga:newGlobals()
_:expect(#new).is(0)
end)
I wonder if it should auto-snap. We’ll consider that. For now, let’s make this work.
GlobalAnalyzer = class()
function GlobalAnalyzer:init()
self.snap = {}
end
function GlobalAnalyzer:copyGlobals()
local g = {}
for k,v in pairs(_G) do
g[k] = v
end
return g
end
function GlobalAnalyzer:snapshot()
self.snap = self:copyGlobals()
end
function GlobalAnalyzer:newGlobals()
local new = {}
local now = self:copyGlobals()
local snap = self.snap
for k,v in pairs(now) do
if snap[k] == nil then
table.insert(new,k)
end
end
table.sort(new)
self.snap = now
return new
end
I’ve decided that the newGlobals method puts the current list into the snapshot, so it’s always notching forward. We’ll see whether we like it.
The test runs.
Now to use it. As soon as I start, I realize that I want snapshot to happen on creation, and I want a method to print the news.
Using that, I decide I want it to provide a single string, not separate prints.
function GlobalAnalyzer:printNewGlobals()
result = "New Globals\n\n"
for i,k in ipairs(self:newGlobals()) do
result = result..k.."\n"
end
print(result)
end
Here’s the report with the princess sitting at the game start:
New Globals
Bus
CodeaTestsVisible
Console
DevMap
GA
OperatingMode
Runner
TileLock
_TileLock
_tweenDelay
arb
items
n
result
tunePlaying
tweenTime
tweentime
After playing the game a bit, I find no more than these. These are, of course of varying interest. The capitalized ones don’t surprise me, they are things created after the top of the program, where I put the creation of the GA. We could move that down, or save the result and refresh, whatever.
But these _ ones I think we can do better.
Adding it in the locals in the dungeon test should make the tile lock one go away:
local dungeon
local _bus
local _TileLock
Same with _tweenDelay? Same kind of thing:
_:before(function()
_TileLock = TileLock
TileLock = false
_tweenDelay = tween.delay
tween.delay = fakeTweenDelay
_Runner = Runner
_bus = Bus
Bus = EventBus()
Runner = GameRunner()
Runner:getDungeon():createTiles(Runner.tileCountX, Runner.tileCountY)
end)
Same fix, make a local in that tab.
No doubt same with _bus … but I don’t find it. I’ll move on for now, chase the low-hanging ones for now.
For arb, I found a test that didn’t say local. Looking for items, I found it in creating decor.
n. That should be interesting. I’ll search for n = and such.
function MapCell:neighbors()
local near = {}
for i,nxy in ipairs(self:neighborIndices()) do
n = self:getLegalCell(self.x+nxy.x, self.y+nxy.y)
if n then
table.insert(near,n)
end
end
return near
end
Clever boy.
result. OK this one is amusing:
function GlobalAnalyzer:printNewGlobals()
result = "New Globals\n\n"
for i,k in ipairs(self:newGlobals()) do
result = result..k.."\n"
end
print(result)
end
The GA did it! That made me LOL.
tunePlaying is semi interesting, it is a shared value in testing the music stuff. Can be local to the test file, I think.
Now tweenTime and tweentime. One is a typo. I think the other should be local to the tests. Let’s run again and see how we’re doing.
New Globals
Bus
CodeaTestsVisible
Console
DevMap
GA
OperatingMode
Runner
TileLock
So that’s nice. This time _bus didn’t come out. Maybe I did it and forgot to write down that I did it.
Anyway, we found a few things. I am still tempted to plug in the code that prevents us from creating new globals, but it crashed Codea because he’s accessing a nil flag.
Turning it on, I get this:
Main:109: attempt to write to undeclared variable lines
stack traceback:
[C]: in function 'error'
Main:38: in metamethod 'newindex'
Main:109: in function 'splitStringToTable'
Player:154: in method 'inform'
Inventory:254: in method 'informObject'
Inventory:245: in method 'informObjectAdded'
Inventory:151: in method 'add'
Inventory:232: in method 'addToInventory'
Decor:151: in method 'giveItem'
Decor:101: in method 'actionWith'
Player:261: in local 'action'
TileArbiter:27: in method 'moveTo'
Tile:109: in method 'attemptedEntranceBy'
Tile:386: in function <Tile:384>
(...tail calls...)
Player:205: in method 'moveBy'
Player:144: in method 'executeKey'
Player:199: in method 'keyPress'
GameRunner:440: in method 'keyPress'
Main:77: in function 'keyboard'
function splitStringToTable(str)
local lines = {}
for s in str:gmatch("[^\r\n]+") do
table.insert(lines, s)
end
return lines
end
Needed that local. I think Codea crashed though. So I cannot leave that enabled. Codea reliably crashes with that global check running, when I leave the Codea screen and come back to it. Codea comes back, not running the program, but instead at its startup screen.
There is another way to deal with global trapping, that involves reporting the result but letting it happen. I’ll look into trying that. For now, I’ll turn that feature back off and commit: Removed spurious globals.
Let’s sum up, I’m hungry.
Summary
We did some nice, if not strictly necessary work on the Making program, supporting nested comments. More a proof that we can do it than anything interesting.
We improved the handling of the patterns, and learned a bit about how best to test them, storing the patterns in functions with the objects that need them, and calling those functions to be sure that the test and the real code get the same pattern.
And, because we were thinking about tools to help make our programs better, we did a quick little tool to check global usage. We’ll see about extending that to log results or something to make it more useful. Or not.
All in all, some practice making tools. Always fun.
See you next time!