XST 34: Happy New Year!
Some HNY thoughts, and more on the function as an element idea. Joy, philosophy, code. What’s not to like?
I hope that the new year brings us all good things. The last couple of years have been pretty rough, but there is still joy to be found, I want to believe. I have to believe. For me, and perhaps for some of my readers, that will come as what GeePaw Hill calls “geek joy”, the sheer pleasure of building software well.
There are more important kinds of joy, of course. I might argue that they all involve building, making something, whether it is a relationship, a family, or a birdhouse. Wherever your joy is, I hope that you will seek it, work at it, and begin to bring a bit of that joy into your life, and the lives of those around you.
For me, one of my ongoing pleasures is in discovering interesting things in programming and related topics, and sharing them with you. Today, I plan to move forward with the notion of functions as first-class members of XST sets. Let’s get started.
The Tentative Plan
Frequent readers know that while I invariably have a plan, it is almost invariably not what happens. One possibility is that I’m a terrible planner. My own belief is that the material we work with takes part in the execution of our plans, and that its “plans” are different from ours, so that what happens isn’t what we planned.
One response to the world resisting our plan is to double down, bear down, dig down and push on through with our plan. All too often, that’s what our corporate um leaders expect and demand. This is rarely the best idea.
A fundamental notion of lower-case agility is to move quickly in response to events. A fundamental notion of upper-case Agility is to develop software in such a way that we can in fact respond rapidly to events. We do that with supporting tests, with well-factored code, and with the greatest skill we can muster.
I mention this as a New Year’s reminder, and also because I figure today’s plan, like so many others, may be adjusted, modified, or shattered, by events. But I have a good feeling about it, actually. We’ll see. The plan is:
First, allow our XFunction object to apply more than one function to the records as it produces them. Second, build another XData subclass that applies functions, not to the current set as XFunction does, but to sets contained within the current set.
Huh. My plan already encounters a problem. What a terrible planner I must be. The issue is that the XFunction does not know the name of the field that the function will create. That value is returned as the second result when the function is called. This has already caused our hasAt a bit of difficulty:
function XFunction:hasAt(element,scope)
if self.set:hasAt(element,scope) then return true end
local value,name = self:evaluate()
return tostring(value)==element and (name or "fcn") ==scope
end
The hasAt function has to answer the question of whether a specific value occurs at a specific scope. Since our function elements are “virtual”, if we don’t find a concrete scope-value pair in the data, we call the function to see whether it happens to match the desired scope and element.
If we have more than one function, I guess we’ll just have to run them all until we find out whether any of them match up.
Maybe not too bad. I’m glad we had this little chat. Our plan will not fall, yet. Let’s get to it.
Many Functions
We’ll begin with a test. I think I can enhance an existing one. Here are the two we have now:
_:test("Function set", function()
local S = XSet:fromTable{a=5,b=50}
local F = XSet:withFunction(S,function(r)
return r:at("a") + r:at("b"), "total"
end)
local output = ""
for value,name in F:elements() do
output = output..name.."="..value.."\n"
end
print(output)
_:expect(F:hasAt("5","a")).is(true)
_:expect(F:hasAt("50","b")).is(true)
_:expect(F:hasAt("55","total"),"total ok").is(true)
_:expect(F:hasAt("22","total"),"total wrong").is(false)
end)
_:test("Function set auto name", function()
local S = XSet:fromTable{a=5,b=50}
local F = XSet:withFunction(S,function(r)
return r:at("a") + r:at("b")
end)
local output = ""
for value,name in F:elements() do
output = output..name.."="..value.."\n"
end
print(output)
_:expect(F:hasAt("5","a")).is(true)
_:expect(F:hasAt("50","b")).is(true)
_:expect(F:hasAt("55","fcn"),"fcn ok").is(true)
_:expect(F:hasAt("22","fcn"),"fcn wrong").is(false)
end)
We are testing two cases, the one where the function does what it’s supposed to and returns a name as well as a value, and one where it doesn’t, and the XFunction supplies one, iin this case “fcn”. Plan change again: we’ll have to generate more than one unique name. (Sub-concern: how can the user possibly know what name has been generated? I suggest that they can’t, and the generated name is just there to ensure that we don’t violate the laws of math or man.)
The protocol withFunction needs to be improved. As a first cut, let’s arrange that withFunction can accept a single function, or an array of functions.
Possible plan change: Maybe we should instead have a method on an XSet, something like addDynamicElement that can add a function to any set at any time. We don’t currently ever change the underlying XData-subclass member of an XSet, but at least in some cases we clearly could.
The present implementation of withFunction doesn’t care what kind of set it goes on top of, it just generates all the elements and then adds in the function element. (Soon to be function elements.)
We’ll let that idea mellow for a bit, and write a new test. The existing ones show some history and it seems better to me to create a new one. I could be wrong.
_:test("Multiple Function Elements", function()
local S = XSet:fromTable{a=5,b=50}
local sum = function(r)
return r:at("a") + r:at("b")
end
local prod = function(r)
return r:at("a") * r:at("b")
end
local F = XSet:withFunction{sum,prod}
_:expect(F:hasAt("55","sum")).is(true)
_:expect(F:hasAt("250","prod")).is(true)
end)
This will explode in some horrible way having to do with an array where we expected a function.
9: Multiple Function Elements -- XData:189: attempt to call a nil value (method 'hasAt')
I’m not even going to try to explore whether that’s “expected”. I know the code is wrong and intend to fix it. Remind me to discuss predicting the error.
He should have checked! See below, and the Summary!
The problem starts here:
function XFunction:init(Set, Function)
self.set = Set
self.fcn = Function
end
We’ll want to distinguish a single function from an array (presumably of functions).
function XFunction:init(Set, FunctionOrArray)
self.set = Set
if type(FunctionOrArray) == "table" then
self.fcn = FunctionOrArray
else
self.fcn = {FunctionOrArray}
end
end
Now we know unconditionally that we have an array of functions to deal with.
The rest of the code needs to be brought into line. Here’s the hasAt:
function XFunction:evaluate()
local value,name = self.fcn(self.set)
return tostring(value),name or "fcn"
end
function XFunction:hasAt(element,scope)
if self.set:hasAt(element,scope) then return true end
local value,name = self:evaluate()
return tostring(value)==element and (name or "fcn") ==scope
end
(We still have that split personality about values that are numbers. Should deal with that soonish.)
Here we need to try all the functions, which means that evaluate will need to be passed the particular one we are checking.
function XFunction:evaluate(f)
local value,name = f(self.set)
return tostring(value),name or "fcn"
end
function XFunction:hasAt(element,scope)
if self.set:hasAt(element,scope) then return true end
for _i,f in pairs(self.fcn) do
local value,name = self:evaluate(f)
if tostring(value)==element and (name or "fcn")==scope then return true end
end
return false
end
I rather expect this to pass. I wonder why I’m wrong.
7: Function set -- XData:81: XData:188: attempt to call a nil value (local 'f')
8: Function set auto name -- XData:97: XData:188: attempt to call a nil value (local 'f')
9: Multiple Function Elements -- XData:200: attempt to call a nil value (method 'hasAt')
OK, this is semi-interesting. The first two errors are there because I’ve not yet fixed elements, so that’s no surprise in retrospect. But the final error is the same as before, which makes me wonder whether there’s a typo or something.
Line 200 is the first line of this:
function XFunction:hasAt(element,scope)
if self.set:hasAt(element,scope) then return true end
for _i,f in pairs(self.fcn) do
local value,name = self:evaluate(f)
if tostring(value)==element and (name or "fcn")==scope then return true end
end
return false
end
What have we stuffed into XFunction’s “set” member?
Ah. Nothing. The withFunction requires the base set as an input, and I didn’t provide it. I’ll have to do this:
local F = XSet:withFunction(S,{sum,prod})
_:expect(F:hasAt("55","sum")).is(true)
_:expect(F:hasAt("250","prod")).is(true)
Now I expect that test to run, and the other two will still fail.
9: Multiple Function Elements -- Actual: false, Expected: true
Well it didn’t blow up, I suppose that can be seen as progress.
I truly wish that access to these things was easier. It’s a problem endemic to set theory but I’m sure I could have better workarounds.
I think I have to make elements work so that I can find out what is in there. Maybe just a bit of tracing.
Oh silly me … I didn’t return the names. Test is wrong.
_:test("Multiple Function Elements", function()
local S = XSet:fromTable{a=5,b=50}
local sum = function(r)
return r:at("a") + r:at("b"), "sum"
end
local prod = function(r)
return r:at("a") * r:at("b"), "prod"
end
local F = XSet:withFunction(S,{sum,prod})
_:expect(F:hasAt("55","sum")).is(true)
_:expect(F:hasAt("250","prod")).is(true)
end)
Test 9 is green. Now for the elements:
function XFunction:elements()
-- return iterator
return coroutine.wrap(function()
for value,field in self.set:elements() do
coroutine.yield(value,field)
end
local result,name = self:evaluate()
coroutine.yield(result,name)
end)
end
Same change:
function XFunction:elements()
-- return iterator
return coroutine.wrap(function()
for value,field in self.set:elements() do
coroutine.yield(value,field)
end
for _i,f in pairs(self.fcn) do
local result,name = self:evaluate(f)
coroutine.yield(result,name)
end
end)
end
Tests are green. I’d commit if I could.
Let’s do a bit of a retrospective, see where we stand.
Specting Retro
This is working much as one would like but there are issues.
Probably the largest is the necessity for the function provided to return the field name (scope). It’s just not a natural thing to do: we do return multiple results sometimes, but not often enough. So it would be better to have a way of expressing these functions including the scope to be used.
Second, the calling sequence to withFunction is bizarre, with a second argument that may or may not be an array of functions.
Relatedly (Second.5?), we can’t create an array of functions with inline code in a readable fashion, so we need something better.
Inside the XFunction code, it would be nice if the function table were, not an array indexed by integers, but a hash table with names. That would help resolve both concerns, and would obviate the need for the magic names. Let’s recast the tests. I think I’ll take it down to just the one, which will now cover the only case where we used to have three:
_:test("Multiple Function Elements", function()
local S = XSet:fromTable{a=5,b=50}
local functions = {}
functions.sum = function(r)
return r:at("a") + r:at("b")
end
functions.prod = function(r)
return r:at("a") * r:at("b")
end
local F = XSet:withFunction(S,functions)
_:expect(F:hasAt("55","sum")).is(true)
_:expect(F:hasAt("250","prod")).is(true)
end)
Here we are now providing a table, functions of named functions. Run the test. It fails for not finding the names, because my functions only return the value, not the name. Fix the code:
XFunction = class(XData)
function XFunction:init(Set, FunctionTable)
self.set = Set
self.fcn = FunctionTable
end
function XFunction:evaluate(f)
local value,name = f(self.set)
return tostring(value)
end
function XFunction:hasAt(element,scope)
if self.set:hasAt(element,scope) then return true end
for name,f in pairs(self.fcn) do
local value = self:evaluate(f)
if tostring(value)==element and (name or "fcn")==scope then return true end
end
return false
end
function XFunction:elements()
-- return iterator
return coroutine.wrap(function()
for value,field in self.set:elements() do
coroutine.yield(value,field)
end
for name,f in pairs(self.fcn) do
local result = self:evaluate(f)
coroutine.yield(result,name)
end
end)
end
Test. The final one works. The other two, do not, but I rather expected that:
7: Function set -- XData:81: XData:204: bad argument #1 to 'for iterator' (table expected, got function)
8: Function set auto name -- XData:97: XData:204: bad argument #1 to 'for iterator' (table expected, got function)
They’re not passing in an array of named functions. One of those tests does a print, which I need until I get a better idea. I’ll move that into the last test and remove the other two as obsolete.
_:test("Multiple Function Elements", function()
local S = XSet:fromTable{a=5,b=50}
local functions = {}
functions.sum = function(r)
return r:at("a") + r:at("b")
end
functions.prod = function(r)
return r:at("a") * r:at("b")
end
local F = XSet:withFunction(S,functions)
_:expect(F:hasAt("55","sum")).is(true)
_:expect(F:hasAt("250","prod")).is(true)
local output = ""
for value,name in F:elements() do
output = output..name.."="..value.."\n"
end
print(output)
end)
Now let’s deal with that loop. I do want to test the elements function, and this is the only check for it. How about this:
_:test("Multiple Function Elements", function()
local S = XSet:fromTable{a=5,b=50}
local functions = {}
functions.sum = function(r)
return r:at("a") + r:at("b")
end
functions.prod = function(r)
return r:at("a") * r:at("b")
end
local F = XSet:withFunction(S,functions)
_:expect(F:hasAt("55","sum")).is(true)
_:expect(F:hasAt("250","prod")).is(true)
local output = {}
for value,name in F:elements() do
table.insert(output, name.."="..value)
end
_:expect(output).has("prod=250")
_:expect(output).has("sum=55")
_:expect(output).has("a=5")
_:expect(output).has("b=50")
end)
This code should run independent of the order of the output. And … it does.
Let me at least zip up this code, since my Git thingie isn’t working today.
I think this might be sufficient for New Year’s Day. Let’s sum up.
Summary
I wanted to talk about predicting test results.
Straight Pool TDD
My colleague Ted M. Young is very much in favor of what I want to call “straight pool TDD”: He wants us to call our shot on every test run. Will it pass? If it will fail, what will the error be?
This is a Very Good Idea, and I am Not Very Good At It. We saw this morning that I got an error on hasAt that was not really the error I should have expected. But I am in the habit of thinking “test won’t run”, not “test will fail saying this and that”, so that even though I had actually said what kind of error I expected, I just went along with the error I got.
That carelessness came back to bite me later. If I were of a mind to “should” myself, I’d say that I should be more careful. But I am not of that mind. However, I will nonetheless try to be a bit more predictive about how tests will fail.
That said, I think that the prediction idea may become less and less applicable as our application grows in complexity. I mean, please. I’m building this XFunction thing that is plugged in under an XSet thing that implements all the smarts, how can anyone expect me to know just what’s going to happen.
But wait …
Ted might say “perhaps you need a finer-grained test, there, Ron”.
And I, being always open to ideas and criticism, might think for a moment and then say “Oh. I see. I could have tested XFunction directly, just checking its elements and hasAt, and at that point been sure that I could plug it into XSet”.
And Ted and I would both be right. Perhaps I’d do better to test these lower-level objects more directly and then, by way of demonstration, write the higher-level tests that actually show what they can do.
Thanks, Ted, I’m glad we had this little chat. Good idea you had there!
Making It Better
Things today did not go quite according to plan. Quelle surprise. In particular, I made both sides of the “this function should (not) return two results” error. I discovered that there was a better implementation of the multiple function feature, and that that implementation actually provided for a better interface for the user-programmer as well.
And I think there may be more coming there. There’s a fine line between
functions.sum = function(...)
And
function functions:sum(...)
It might make sense for the user (me) to define a little class with the functions in it. But maybe not. It would take some experimentation.
But we’ve already made the function capability a bit better but in notation and in power, and we’ve actually simplified it. It used to be that you could optionally provide a name, in an awkward way, and our code would then awkwardly try to invent a name for you. Now you have an easy and non-optional way to provide a name, so it’s better for all of us.
There is an interesting issue: what would happen if you were to provide an array of functions rather than a hash table? I think the answer is that you’d get numeric field names. Maybe we should check that out, but for our learning purposes it’s not a high priority.
So, as always … we deviated from the plan … and as usual … we did something better.
That does generally mean that we don’t get as far. Just as, if you’re on a driving vacation and you decide to stop in Baraboo and check out the circus stuff, you may not get to Okoboji quite so soon, but you’ll probably have more fun.
Which brings us back to joy. I enjoy this. I commend to you to do things that you enjoy. And love those around you, and let them love you.
See you next time!