XST 36: Technical Debt?

It’s 6 AM and I have an idea. This could be very good or very bad.

Life’s pretty slow here chez Ron, and while I usually wake up briefly around 6 AM, I generally try to sleep later, so that I’ll be ready for napping during the day. But this morning at 0445 I woke up and started thinking. I fell back asleep, was awakened again by a cat that needed to stand on me, and kept on thinking. I’ve decided to put the thinking down on well, not paper, but on my web site.

Here, before I forget more of them, are this morning’s ideas:

• hasAt has a generic implementation
• maybeHasAt should be exist
• try(method)
• XFunction vs XInnerFunction
• filters
• set types as properties not classes
• morphing set structure when it gets curly

Technical Debt

Yesterday, as it happens, I subtweeted “That’s not technical debt.” What we have today is, in fact, technical debt.

I adhere to Ward Cunningham’s original notion of technical debt, which I would describe as the gap between what we understand about the problem and our solution, and the solution we now see as a better one. Ward describes it here.

Kent Beck’s four rules of simple design include “the code expresses all our ideas about the design”. When that is not the case, we may be looking at technical debt. There are two cases when we have ideas that are not in the code:

1. The ideas came before we wrote the code and we didn’t express them;
2. We have had new ideas and have not expressed them (yet).

The latter is technical debt as I understand it. The former is just code that is not as good as it might be.

Often, people use “technical debt” when they mean “we took a shortcut to get this code out”. Such code generally needs improvement in order to work correctly. That’s not technical debt. It may be a judicious choice to get to market with something that we might get away with, or–much more often in my opinion–it was a mistaken decision in hopes of “going faster”.

Whatever it is, in my book that form of gambling is probably unwise, and it is obfuscation to refer to it as “technical debt”. A better phrase is “we thought we could go faster if we half-assed this part of the system”.

But what we have here, I believe, is real live technical debt.

A Bit More Detail

Let’s see what these ideas mean to me. Mind you, they are now about two hours or less old, so they’re pretty fresh.

hasAt has a generic implementation

All our XData subclasses are required to implement hasAt. But I just realized that `hasAt has a general implementation. We should change from requiring its implementation to allowing it to be overridden.

Maybe hasAt should be exist

Corollary to the above. The standard implementation will use exist, most likely.

try(method)

We want to use an overridden method in XData subclasses if one exists. We can probably implement a try function that can be used as a prefix to all the overridable set operations, rather than hand-code each such check.

XFunction vs XInnerFunction

XInnerFunction class can probably be removed by epplying a suitable function in XFunction. Or maybe it’s the other way around. One class should be able to do both, it seems to me.

Filters

We have functions that are applied “during” the production of set elements. We might do better to have functions to be applied before, during, and after.

Set types as properties not classes

At last some set types, “tuple” comes to mind, might be better expressed as properties, propositions about sets that are known to be true. “tuple” is such a property.

Morphing set structure when it gets curly

The current generic set structure can support sets of sets, more than one element under a given scope, and so on. Most sets that we create are simpler than that. It might be better to assume simplicity and gracefully become more complex only as needed. (That said, the Xeneric structure is quite fast and doesn’t use much more memory than a simpler structure would.)

4AM Ideas

These are 4AM ideas and may not all be entirely baked. But they begin to suggest that we have more complexity in our design than we really need. That complexity is legitimate, in that it was the best way that we–OK, I–saw at the time, but now, quite possibly, we can do better.

If we can, what we have in hand is true technical debt.

Let’s explore some of these ideas.

hasAt

There’s a lot not to like about hasAt. It just answers the question “Given a set S, a scope s and an element e, is e an element of the set S, under scope s?”

I’m about to check this, but I suspect that the only legitimate uses of hasAt are in tests. There may be a few exceptions for efficiency. Let’s look. I’ll put below all the non-test occurrences of hasAt:

function XSet:hasAt(element, scope)
    return self.data:hasAt(element,scope)
end

function XSet:hasntAt(e,s)
    return not self:hasAt(e,s)
end

function XSet:hasElement(valueTable)
    return self:exists(function(rec,_s)
        local hasAll = true
        for scope,value in pairs(valueTable) do
            hasAll = hasAll and rec:hasAt(value,scope)
        end
        return hasAll
    end)
end

function XSet:intersect(anXSet)
    return self:select(function(e,s) return anXSet:hasAt(e,s) end)
end

function XSet:isSubset(other)
    -- return whether self is a subset of other
    -- for every element e,s of self
    --   is e element-s of other
    return self:every(function(e,s) 
        return other:hasAt(e,s) 
    end)
end

function XSet:selectFieldValue(scope,value, index)
    local ix = self:findSelectIndex(scope,value,index)
    if ix then
        print("used index")
        return self:scopeRestrict(ix)
    else
        return self:select(function(record, recordScope)
            return record:hasAt(value,scope)
        end)
    end
end

function XSet:selectFieldValue(scope,value, index)
    local ix = self:findSelectIndex(scope,value,index)
    if ix then
        print("used index")
        return self:scopeRestrict(ix)
    else
        return self:select(function(record, recordScope)
            return record:hasAt(value,scope)
        end)
    end
end

I freely grant that that is more uses than I would have guessed. The reason is that when I think of most of those operations, I think directly about them, not about how they are implemented.

In the above, hasElement is used only in tests. It is a convenience method that I had forgotten about. It’s used like this:

        _:test("Grouping", function()
            local control = {sum={"pay"}, group={"state"} }
            local result = People:stats(control)
            _:expect(result:card(),"result").is(2)
            local foundMI = result:hasElement{state="MI",pay="3000"}
            _:expect(foundMI,"MI").is(true)
            local foundOH = result:hasElement{state="OH",pay="300"}
            _:expect(foundOH,"OH").is(true)
        end)

The method allows determination whether there is an element (record) in the set that includes more than one element-scope pair.

The form of that test, and the form of all the tests that use hasAt is a continuing irritation, as it means that checks require more than one line and the diagnostics are all “got false expected true”, which is less than informative.

Assessment

We’re here to deal with the possibility of a generic implementation of hasAt. Let’s look at all the implementations of hasAt, a slightly different view from the one above:

function XSet:hasAt(element, scope)
    return self.data:hasAt(element,scope)
end

function XData:hasAt()
    error("Must implement hasAt!")
end

function XExpression:hasAt(e,s)
    local data = self:findDataSet(self.xset)
    if data then return data:hasAt(e,s) else return false end
end

function XFunction:hasAt(element,scope)
    if self.set:hasAt(element,scope) then return true end
    for name,f in pairs(self.fcn) do
        local value = self:evaluate(f)
        if tostring(value)==element and (name or "fcn")==scope then return true end
    end
    return false
end

function XGeneric:hasAt(element, scope)
    local tab = self.contents[scope]
    if not tab then return false end
    if type(element) ~= "table" then
        return tab[element] == true
    else
        for e,_true in pairs(tab) do
            if type(e) == "table" and e:equals(element) then
                return true
            end
        end
        return false
    end
end

function XTuple:hasAt(e,s)
    local element = self.contents[s]
    if type(e)~="table" then
        return element and e==element
    else
        return type(e)=="table" and element:equals(e)
    end
end

function CSVLineSet:hasAt(e,s)
    local index = self.labels[s]
    if index == nil then return false end
    return self.fields[index] == e
end

Interesting discovery: CSVSet has no implementation of hasAt. That’s probably a defect, but equally probably no one is ever going to do it.

Also XInnerFunction has no hasAt. I think I mentioned that yesterday with a hand-wave at doing it today. Better make a note: Done. The note, not the feature.

General hasAt

Let’s provide the generic hasAt at the XSet level. I don’t think we need to write additional tests because if we override hasAt in XSet all the tests will be affected. Still, it might be worth adding a hasAt test for XInnerFunction while we are here, if we can.

        _:test("XSet inner functions", function()
            local A = XSet:fromTable({ a="error",{a=5}, {a=50} })
            local f = {}
            f.double = function(r) return r:at("a")*2 end
            local D = A:withInnerFunctions(f)
            local hasA = D:hasAt("error","a")
            _:expect(hasA, "D has error at a").is(true)
            for e,s in D:elements() do
                if type(e)=="string" then
                    -- do nothing
                elseif e:at("a") == "5" then
                    _:expect(e:at("double")).is("10")
                elseif e:at("a") == "50" then
                    _:expect(e:at("double")).is("100")
                end
            end
        end)

I added that first expectation which was sufficient to elicit the error:

10: XSet inner functions -- XData:183: Must implement hasAt!

There’s a problem. This is what happens when you believe a 4 AM idea.

The general implementation of hasAt is quite straightforward. Let me just do it:

function XSet:hasAt(element, scope)
    return self:exists(function(e,s) return self:EQ(element,e) and self:EQ(scope,s) end)
end

function XSet:EQ(x,y)
    if self:isXSet(x) and self:isXSet(y) then
        return x:equals(y)
    else
        return x==y
    end
end

This implementation runs all the tests. The EQ is just there because we have not … oh my … because we have not overridden == in XSet. Why haven’t we just done that?

If we can, we can just write this:

function XSet:hasAt(element, scope)
    return self:exists(function(e,s) return element==e and scope==s end)
end

If we have not yet done == on XSets, some tests fail:

9: Curly Union sets equal -- Actual: false, Expected: true
14: elements set equality -- Actual: false, Expected: true
35: Tuple equal -- Actual: false, Expected: true

We implement --eq on XSet:

function XSet:__eq(something)
    if self:isXSet(something) then
        return self:equals(something)
    else 
        return false
    end
end

Let’s inline equals and then remove it, so that everyone uses == instead:

function XSet:__eq(something)
    if self:isXSet(something) then
        return self:isSubset(something) and something:isSubset(self)
    else 
        return false
    end
end

This change does wonders. It lets us change tests like this:

        _:test("set equality", function()
            local one = XSet():addAt("Ron","first"):addAt("Jeffries","last")
            local two = XSet():addAt("Ron","first"):addAt("Perlman","last")
            _:expect(one:equals(one)).is(true)
            _:expect(one:equals(two)).is(false)
        end)

To this:

        _:test("set equality", function()
            local one = XSet():addAt("Ron","first"):addAt("Jeffries","last")
            local two = XSet():addAt("Ron","first"):addAt("Perlman","last")
            _:expect(one).is(one)
            _:expect(one).isnt(two)
        end)

After some editing, the tests are green. Commit: Implemented XSet==XSet and applied it everywhere.

Now there’s this:

function XSet:__eq(something)
    if self:isXSet(something) then
        return self:isSubset(something) and something:isSubset(self)
    else 
        return false
    end
end

Permit me this:

function XSet:__eq(something)
    return self:isXSet(something) and self:isSubset(something) and something:isSubset(self)
end

Green. Commit: simplify XSet:__eq

Let’s lift our heads for a moment.

Retrospective

No plan survives contact with the code, a famous programming general once said, and they were right. We have, however, implemented a generic hasAt, which obviates the need for our XData guys to implement it. However … at the moment, if they do implement it, their private hasAt will not be used:

function XSet:hasAt(element, scope)
    return self:exists(function(e,s) return element==e and scope==s end)
end

Let’s modify this so that if our self.data implements hasAt, we use it:

function XSet:hasAt(element, scope)
    if self.data.hasAt then
        return self.data:hasAt(element,scope)
    else
        return self:exists(function(e,s) return element==e and scope==s end)
    end
end

I expected no errors from this but in fact my prior test pops up to remind me:

10: XSet inner functions -- XData:183: Must implement hasAt!

Ah. A bit more exploration than I’d like tells me something that I knew but wasn’t expecting. When a subclass is created, the new class receives all the contents of the superclass: they’re just copied right in.

That means that the subclasses of XData receive XData’s implementation of anything that it implements, in this case:

function XData:hasAt()
    error("Must implement hasAt!")
end

Since XSet is checking to see whether its data element implements hasAt, XData cannot implement it at all. If I remove that method the tests should go back to green, and XInnerFunction will again use the generic hasAt.

Yes, green: Commit: remove “must implement” hasAt from Xdata to permit XSet to check for overrides.

We now have a few other concerns in XData:

XData = class()

function XData:__tostring()
    return "XData???"
end

function XData:init()
    error("Must implement init!")
end

function XData:card()
    return nil
end

function XData:elementProject()
    return nil
end

function XData:elements()
    error("Must implement elements!")
end

The final one is correct. We will unconditionally ask the data to produce the elements:

function XSet:elements()
    return self.data:elements()
end

What about card and elementProject? Let’s see how those are used.

The XTuple class implements both of those:

function XTuple:card()
    return #self.contents
end

function XTuple:elementProject(scope)
    return self.contents[scope]
end

It also implements hasAt. Are we checking for nil return from those two methods in XSet?

function XSet:card()
    return self.data:card() or self:reduce(0, function(r,e,s) return r+1 end)
end

function XSet:elementProject(scope)
    return self.data:elementProject(scope) or self:elementProjectLong(scope)
end

Now we have two conventions for determining whether an XData subclass has support for an operation or not. In hasAt we look to see if it is implemented. In card and elementProject, we call the function and if the result is nil, we do it the long way. In the case of elementProject, that’s actually a bad idea, because when tuple says “no” by returning nil, the answer is no and searching won’t help.

So let’s go to the same convention for them all, checking the instance for nil (I think we’ll improve that in a moment, or at least consider it).

function XSet:card()
    return self.data.card and self.data:card() or self:reduce(0, function(r,e,s) return r+1 end)
end

This really ought to break something but I bet it doesn’t. I hope it does, but don’t think it will. And it doesn’t. Maybe that’s OK. I remove the XData:card entirely and everything works.

We can be certain that card gets the right answer. What we cannot be certain of is that it used the provided fast implementation in the one case of XTuple. Well, the code tells me that we did. There’s no test for it.

Let’s do elementProject as well.

function XSet:elementProject(scope)
    return self.data.elementProject and self.data:elementProject(scope) or self:elementProjectLong(scope)
end

Tests all run. Commit: card and elementProject check data element for corresponding method and otherwise use default long-form method. (No tests to be sure short methods are used.)

Let’s figure out a way to check those methods for being used. We’d like to do that without slowing them down in general.

OK, I have an evil idea. What if we monkey-patch them in the tests to see if they are called?

Here’s the Tuple test as modified:

        _:test("People Tuple", function()
            local csv = XSet:on(CSVSet(CSVnames, CSVdata))
            local tab = {}
            for e,s in csv:elements() do
                table.insert(tab,e)
            end
            local PT = XSet:tuple(tab)
            local oldCard = PT.card
            local gotThere = false
            PT.card = function() gotThere = true; return oldCard(PT) end
            _:expect(PT:card()).is(500)
            _:expect(gotThere,"card not called").is(true)
            PT.card = oldCard
            _:expect(PT:is_a(XSet),"is XSet").is(true)
            local restrictor = XSet()
            for _i,state in ipairs({"MI", "AK", "KY", "NE", "NY"}) do
                restrictor:addAt(XSet():addAt(state,"state"),NULL)
            end
            local restricted = PT:restrict(restrictor) -- was csvRestrict
            _:expect(restricted:card()).is(69)
            local j = restricted:elementProject(2)
            _:expect(j:elementProject("first_name")).is("Josephine")
        end)

I monkey-patched card in PT to set my local flag and call the old function that was there. The test runs, so we know that XSet called `card in the XTuple.

Horrid, but I don’t have a better idea. Commit: test for Tuple:card now monkey patches to ensure correct method called.

OK, we can do that for EP as well. Our starting test:

        _:test("Tuple", function()
            local set = XSet():addAt("Jeffries","last")
            local equ = XSet():addAt("Jeffries","last")
            local T = XSet:tuple{"a","b","c","d",set}
            _:expect(T:card()).is(5)
            local c  = T:elementProject(3)
            _:expect(c).is("c")
            _:expect(T:hasAt("b",2)).is(true)
            _:expect(T:hasAt("c",2)).is(false)
            _:expect(T:hasAt(set,5),"identical").is(true)
            _:expect(T:hasAt(equ,5),"equal").is(true)
            local t = {}
            for e,s in T:elements() do
                t[s]=e
            end
            _:expect(t[1]).is("a")
            _:expect(t[2]).is("b")
            _:expect(t[3]).is("c")
            _:expect(t[4]).is("d")
        end)

With monkey-patching, that becomes:

        _:test("Tuple", function()
            local set = XSet():addAt("Jeffries","last")
            local equ = XSet():addAt("Jeffries","last")
            local T = XSet:tuple{"a","b","c","d",set}
            _:expect(T:card()).is(5)
            local oldEP = T.elementProject
            local done = false
            T.elementProject = function(set,scope)
                done = true
                return oldEP(T, scope)
            end
            local c  = T:elementProject(3)
            T.elementProject = oldEP
            _:expect(done).is(true)
            _:expect(c).is("c")
            _:expect(T:hasAt("b",2)).is(true)
            _:expect(T:hasAt("c",2)).is(false)
            _:expect(T:hasAt(set,5),"identical").is(true)
            _:expect(T:hasAt(equ,5),"equal").is(true)
            local t = {}
            for e,s in T:elements() do
                t[s]=e
            end
            _:expect(t[1]).is("a")
            _:expect(t[2]).is("b")
            _:expect(t[3]).is("c")
            _:expect(t[4]).is("d")
        end)

Wow that’s nasty. But how else can I really know that the specific overriding function is called?

There is another instance where I’d like to do this. Whenever I run the tests, they print “used index” because of this:

function XSet:selectFieldValue(scope,value, index)
    local ix = self:findSelectIndex(scope,value,index)
    if ix then
        print("used index")
        return self:scopeRestrict(ix)
    else
        return self:select(function(record, recordScope)
            return record:hasAt(value,scope)
        end)
    end
end

That happens at the end of this test:

        _:test("indexedRestrict", function()
            local r1 = XSet():addAt("Ron","name"):addAt("MI","state")
            local r2 = XSet():addAt("Bill","name"):addAt("VA","state")
            local r3 = XSet():addAt("Pat","name"):addAt("MI","state")
            local r4 = XSet():addAt("Chet","name"):addAt("GA","state")
            local people = XSet:tuple{r1,r2,r3,r4}
            local mi = XSet():addAt(1,NULL):addAt(3,NULL)
            local va = XSet():addAt(2,NULL)
            local ga = XSet():addAt(4,NULL)
            local indexes = XSet():addAt(mi,"MI"):addAt(va,"VA"):addAt(ga,"GA")
            local stateSet = XSet():addAt(indexes,"state")
            local allIndexes = XSet():addAt(stateSet,"INDEXES")
            local descoped = people:scopeRestrict(mi)
            _:expect(descoped:card()).is(2)
            local desiredIndex = indexes:elementProject("MI")
            local usedIndex = people:scopeRestrict(desiredIndex)
            _:expect(usedIndex).is(descoped)
            local selected = people:selectFieldValue("state","MI", allIndexes)
            _:expect(selected).is(descoped)
        end)

That last call to selectFieldValue is able to use the index. Here, we can trap the call to scopeRestrict.

First, I’d better commit: tests for card and elementProject monkey patch to ensure overridden methods called.

Now let’s see if we can patch that last bit in the test above.

            local usedIndex = people:scopeRestrict(desiredIndex)
            _:expect(usedIndex).is(descoped)
            local oldSR = people.scopeRestrict
            local done = false
            people.scopeRestrict = function(A,B)
                done = true
                return oldSR(A,B)
            end
            local selected = people:selectFieldValue("state","MI", allIndexes)
            people.scopeRestrict = oldSR
            _:expect(done,"index not used").is(true)
            _:expect(selected,"wrong result").is(descoped)

Tests green. Remove that print and commit: use monkey patch to test that index is used on selectFieldValue.

Time for another look back.

Retrospective II

The general approach to a general implementation for hasAt seems to me to be good. The specific technique for detecting an override has led to some duplication:

function XSet:card()
    return self.data.card and self.data:card() or self:reduce(0, function(r,e,s) return r+1 end)
end

function XSet:elementProject(scope)
    return self.data.elementProject and self.data:elementProject(scope) or self:elementProjectLong(scope)
end

function XSet:hasAt(element, scope)
    if self.data.hasAt then
        return self.data:hasAt(element,scope)
    else
        return self:exists(function(e,s) return element==e and scope==s end)
    end
end

The first two are more obviously duplication than the third, perhaps. Let’s make them more similar.

function XSet:card()
    return self.data.card and self.data:card() 
        or self:reduce(0, function(r,e,s) return r+1 end)
end

function XSet:elementProject(scope)
    return self.data.elementProject and self.data:elementProject(scope) 
        or self:elementProjectLong(scope)
end

function XSet:hasAt(element, scope)
    return self.data.hasAt and self.data:hasAt(element,scope) 
        or self:exists(function(e,s) return element==e and scope==s end)
end

Each of these checks self.data to see whether it has a function and calls that function if it exists, otherwise returning a default result.

So we could write this instead:

function XSet:hasAt(element, scope)
    return self:try("hasAt")
        or self:exists(function(e,s) return element==e and scope==s end)
end

function XSet:try(method, ...)
    return self.data.method and self.datammethod(self.data, ...)
end

The tests are green. The try method name is weak, but I don’t have a better idea yet. Let’s use it in the other two cases:

function XSet:elementProject(scope)
    return self:try("elementProject",scope)
        or self:elementProjectLong(scope)
end

That works.

function XSet:card()
    return self:try("card") 
        or self:reduce(0, function(r,e,s) return r+1 end)
end

Also works. However, upon looking at the horrendous mistakes in the try method above, I am inclined to remove it. It can’t ever trigger as written, and so everything defaults.

A quick attempt to make it right gives me errors:

function XSet:try(method, ...)
    return self.data[method] and self.data[method](self.data, ...)
end

I’m not sure about the … so I’ll do this:

function XSet:try(method, a,b,c,d)
    return self.data[method] and self.data[method](self.data, a,b,c,d)
end

Still borked. Clearly this is too fancy for my shirt. Remove it with a revert.

Let’s push down to another level of retro

Retrospective III

OK, there’s duplication there, but it isn’t as easy to get rid of as I had thought it would be. Fine. Maybe I’ll try again when I’m more sharp, should that day ever come. The thing is to realize when you’re being too clever and cut it the heck out.

That aside, we have some improvements. We have a general implementation of hasAt that can be overridden but need not be, relying on elements. That simplified XData a bit, and, together with the different format for checking for overrides, allows XData subclasses more flexibility in what they do or do not implement.

Thinking about all this somehow made me realize that I could implement __eq on XSet, which allows us to write A==B rather than A:equals(B). That simplified a lot of tests. It didn’t reduce the number of lines, it just made the tests more clear and easier to write.

It’s 0930, and I started at 0600, so it might be about time to stop. Let’s look at the original plan:

• hasAt has a generic implementation
• maybeHasAt should be exist
• try(method)
• Xfunction vs XInnerFunction
• filters
• set types as properties not classes
• morphing set structure when it gets curly

We did the first two, which are really just one. And we tried the third, but the first two attempts were bogus, so I belayed that idea.

I do think that these two belong together, and are possibly worth looking at:

• XFunction vs XInnerFunction
• filters

It seems to me that the functions we use for these two classes are similar. XFunction has a set of functions that it calls after generating its elements. It also has a specialized hasAt. Do we need that?

Meanwhile XInnerFunction has a single built-in function that it applies to each element before it is generated.

function XInnerFunction:elements()
    -- return iterator
    return coroutine.wrap(function()
        for record, scope in self.set:elements() do
            if record.withFunctions then record = record:withFunctions(self.fcn) end
            coroutine.yield(record, scope)
        end
    end)
end

That code is odd, isn’t it? What it’s doing is trying to be sure that it doesn’t call withFunctions on something that wouldn’t understand. We could replace that with a check for isXSet:

function XInnerFunction:elements()
    -- return iterator
    return coroutine.wrap(function()
        for record, scope in self.set:elements() do
            if XSet:isXSet(record) then record = record:withFunctions(self.fcn) end
            coroutine.yield(record, scope)
        end
    end)
end

That’s better, more legitimate. Commit: modify XInnerFunction:elements to use XSet:isXSet.

Now what about that hasAt, compared to its more general implementation?

function XFunction:hasAt(element,scope)
    if self.set:hasAt(element,scope) then return true end
    for name,f in pairs(self.fcn) do
        local value = self:evaluate(f)
        if tostring(value)==element and (name or "fcn")==scope then return true end
    end
    return false
end

That isn’t even current. The stuff with fcn no longer applies: Remove and commit.

function XFunction:hasAt(element,scope)
    if self.set:hasAt(element,scope) then return true end
    for name,f in pairs(self.fcn) do
        local value = self:evaluate(f)
        if tostring(value)==element and name==scope then return true end
    end
    return false
end

This code, in either form, calls hasAt on the inner set, and then iterates the functions to see if they match, if the set doesn’t respond favorably.

Now if the inner set has a fast hasAt, we’ll lose some time if we allow the generic to use it. The only set types with a fast hasAt are Tuple and CSVLineSet:

function XTuple:hasAt(e,s)
    return self.contents[s] == e
end

So if we ever applied functions to a Tuple, we’d lose out. Is the added complexity worth keeping? Well, I think it might be. In principle (but not current practice) a conventional relation, consisting of a rack of records, would be well-represented as a tuple, because we could index it by record number. We aren’t doing that, so arguably we should remove this capability as unused. But maybe someday, and it does work.

And CSVLineSet checks its label table to see if the scope you’re asking for exists. So that saves time over iterating the record’s fields.

We’ll let this code in XFunction live. It works, it’s tested, and it saves a bit of time. Judgment call but sunk cost does provide some benefit and the added complexity isn’t visible much of anywhere.

The larger questions

Some larger question are whether there is some more general implementation of XFunction and XInnerFunction that would bring the two of them together, and whether we could better represent some set types with simple properties. Probably a smaller question is whether all that might let us make our base set structure simpler and then expand it somehow dynamically in the rare case where the set in question has the more complex structure that we call “curly”.

A simple set structure might be an elementary hash table from simple scope to element, and only if we tried to add another element at the same scope, or a scope that was not simple, would we switch over to our more generic structure.

Fact is, we aren’t supporting sets as scopes even now:

function XGeneric:addAt(element, scope)
    assert(scope~=nil,"nil scope "..tostring(element))
    if self:hasAt(element,scope) then return end
    if not self.contents[scope] then
        self.contents[scope] = {}
    end
    self.contents[scope][element] = true
    return self
end

We have had no need for the Set As Scope capability and it is not implemented. This is not “technical debt”, by the way. It is a decision about what functionality to implement, so it is essentially a product design decision. Yes, we saved time and effort by not implementing it. Great. But it’s not technical debt.

We’ve been around this loop before. Yes, it is probably possible to implement our usual XSets a bit more efficiently in space and time, but not much. Allowing for multiple elements at the same scope isn’t costly, and it deals nicely with the fact that many sets default to have NULL as the scope for their elements.

I think there may be more benefit to detecting when a set is or is not a tuple, and treating that as a property that a set could maintain or remove depending on what happens. But since all sets are essentially immutable after creation, there’s not much downside to actually choosing to create an XTuple if we want one. Automating the transition would have little payoff, if any.

I think we’ll sum up.

Summary

Interesting morning.

Retrospectives went right to action …

Did you notice that every time I started to do a retrospective, I found something that could be improved? I hope so. But did you notice that I didn’t write it on a card to be done later? Instead I just went ahead and did it. I went directly from reflection to discovery to improvement. This is often possible with code. Probably less possible with human process.

Impact less than I had thought …

I had expected to remove a bit more technical debt than was actually accomplished. I’m not sure whether to count A==B as technical debt or not. Of course it doesn’t matter: it’s an improvement. But I didn’t see it at the beginning and I do see it now, so I guess technically it is technical debt.

Improving the hasAt situation, as well as card and elementProject are improvements and do count as technical debt payoff, because here again they were the best we knew at the time and we know better now.

However, I was really expecting larger impact on code size than what we got. I suspect there are some discoveries yet to be had, although the code is generally pretty tight. I think there’s a big understanding to be found in the function application / filters notion but it’s not in my mind yet, so I can’t put it in the code.

Apparently these ARE my monkeys …

The monkey-patching idea in tests is a mixed blessing. It does let me detect when a specific method was called, and when doing overrides for optimization, we need to know that. But it is certainly pretty nasty in its current form. Maybe there’s a way to improve that, maybe even to build it into CodeaUnit.

Net net net

So a good morning, a noticeable improvement and simplification in the code, a new albeit unattractive way of writing tests, and more to think about. A perfect morning.

See you next time!

XSet2.zip