FAFO on GitHub

A check of our list of ideas leads to discoveries, to key decisions, and to reflection on the realities of software development. Quite a span, as perhaps it always should be. Interesting article reference included.

Here’s the list from yesterday:

  • Lex underbar ala “total_pay”
  • refactor result method. (I would have to search to find out what this refers to.)
  • Should XSet implement __contains__?
  • Should we use complex scopes like <salary, int>?
  • Improve restrict code?
  • Use reduce in XFlat fields?
  • XRelation - one set of scopes for entire set, save memory?
  • Spill storage - tuples could be easy to store?
  • Better use of map and other itertools?
  • Remember scopes in project operator? (Not sure what this means.)
  • Improve Flat symbol table? Use XST in it?
  • Could a, b, c = Xset return three tuples? (Probably not.)
  • Does re_scope use comprehensions?
  • What about very large sets?
  • Do we want is_null?

Some of these items are quite old. I think I’ll try to remember to put down the date when I write them, because they are often quite obsolete. I might do them without remembering they’re on the list, or things might change enough so that they’re no longer applicable, etc.

I started looking for a test relating to this item:

  • Could a, b, c = Xset return three tuples? (Probably not.) for a related idea. It is passing but, I think for the wrong reason, because (I am sure) subscripting now works, meaning to select an item with the provided scope. KInd of like what a dictionary might do.
    def test_subscripting(self):
        x = XSet.from_tuples((("a", 1), ("b", 2)))
        with pytest.raises(TypeError):
            e, s = x[0]  # must implement getitem for this to work

This revised test passes also, and is more correct:

    def test_subscripting(self):
        x = XSet.from_tuples((("a", 1), ("b", 2)))
        e = x[1]
        assert e == 'a'
        e = x[0]
        assert e is None

Looking further for a place to write a test for the a, b, c idea above, I find that we have one! In fact we have three!

    def test_tuples_from_set(self):
        s = XSet.n_tuple(['a', 'b', 'c'])
        try:
            a, b, c, d = s
        except ValueError:
            pass
        try:
            a, b = s
        except ValueError:
            pass
        a, b, c = s
        a_e, a_s = a
        assert a_e in ['a', 'b', 'c']
        assert a_s in [1, 2, 3]

    def test_tuple_split(self):
        s = XSet.n_tuple(['a', 'b', 'c'])
        (a0, a1), (b0, b1), (c0, c1) = s
        assert a0 in ['a', 'b', 'c']
        assert b0 in ['a', 'b', 'c']
        assert c0 in ['a', 'b', 'c']

    def test_n_tuple_string(self):
        n = XSet.n_tuple(['a', 'b', 'c'])
        output = [f'{e}^{s}' for e, s in n]
        result = ', '.join(output)
        assert result == 'a^1, b^2, c^3'

We can assign from a set! What comes out are tuples, and we must provide as many variables as there are elements in the set. So in the first test, we have to ask for three tuples, not two or four. The test thinks they can come out in arbitrary order, so it is testing with that assumption in mind.

With a set created with n_tuple, however, the set will be produced in numeric order. That is clearly demonstrated in the final test, which is assuming order 1, 2, 3 and gets it. The question in my mind is:

In general, a set has no particular order and when it is iterated, the tuples will be produced in an unpredictable order. Is it a good idea to say, yabbut, an n_tuple, when iterated, will be produced in order 1-n?

I think it is not appropriate. Let’s comment that last test to that effect:

    def test_n_tuple_string(self):
        n = XSet.n_tuple(['a', 'b', 'c'])
        output = [f'{e}^{s}' for e, s in n]
        result = ', '.join(output)
        # This result relies on a fact not in evidence,
        # namely that an n_tuple produces its elements in numeric order.
        # If this fails ... what should we do? I do not know. R: 20240309
        assert result == 'a^1, b^2, c^3'

Commit: Add comment to test that succeeds almost by accident.

I add a new note to my note cards: Decide about n-tuple producing element in scope order (20240309).

Aside:
I wonder. What if I made a little markdown file in the project for notes like this? Would that be a good thing? I could quickly develop the habit of putting the list in there and maintaining it. Or is that the first big step on the road to the depths of deg-ra-day? And, I have these most excellent quad-ruled cards that I love, almost as nice as the custom-printed ones I used to have.

I like the hands-on nature of the cards, writing on them, crossing out, throwing them away when they are all crossed out. I’ll step back from the slippery slope that leads to Jira, at least this time.

Let’s drill down a bit.

The only sensible reason one might have for writing this program would be to support implementation of some end-user program, whether a relational database app, a data analysis app, something like that. (The non-sensible reason, of course, is the one we’re using at the moment. We want to.) An end user will almost certainly want to produce results in a sorted order. What does it mean for a set to be sorted? I believe that the official set theoretic answer is that a sort puts the items (*) in the set into a correspondence with the integers 1-n, where n is the number of elements in the set.

At first blush, one might think, hey, that’s the n-tuple right there. But not so fast. I wrote “items in the set” above, but before I wrote that, I wrote “elements in the set”. What’s the difference? Suppose we had this set:

{ jofirst, adafirst, bethfirst }

When we sort that set, do we want this:

{ ada1, beth2, jo3 }

Or do we want something like this:

{ {adafirst}1, {bethfirst}2, {jofirst}3 }

If we say that sort puts the elements of the input set into correspondence with the integers, then we are committing to the first definition, and if you want the second result, you should have an input set more like this:

{ {jofirst}, {adafirst,} {bethfirst} }

OK. Let’s tentatively say that if a set is known to be an n-tuple, when it is iterated its element-scope tuples will be produced in increasing numeric order. We will leave open for now the question of actually sorting, and the question of when, if ever, the system will know that a set is an n-tuple, aside from its original creation.

I can cross out my new item and change those tests to reflect the new decision. I’ll add a new one as well, to make the point more clear:

    def test_n_tuple_produces_elements_in_order(self):
        s = XSet.n_tuple(['a', 'b', 'c'])
        it = iter(s)
        assert next(it) == ('a', 1)
        assert next(it) == ('b', 2)
        assert next(it) == ('c', 3)

    def test_tuples_from_set(self):
        s = XSet.n_tuple(['a', 'b', 'c'])
        try:
            a, b, c, d = s
        except ValueError:
            pass
        try:
            a, b = s
        except ValueError:
            pass
        a, b, c = s
        a_e, a_s = a
        assert a_e == 'a'
        assert a_s == 1

    def test_tuple_split(self):
        s = XSet.n_tuple(['a', 'b', 'c'])
        (a0, a1), (b0, b1), (c0, c1) = s
        assert a0 == 'a'
        assert b0 == 'b'
        assert c0 == 'c'

    def test_n_tuple_string(self):
        n = XSet.n_tuple(['a', 'b', 'c'])
        output = [f'{e}^{s}' for e, s in n]
        result = ', '.join(output)
        assert result == 'a^1, b^2, c^3'

Commit: Tests showing n-tuples produce elements in scope order.

Observation

It occurs to me that we can convert any XSet to a list, because they are iterable. This test passes:

    def test_n_tuple_as_list(self):
        n = XSet.n_tuple(['a', 'b', 'c'])
        tuples = list(n)
        assert tuples == [('a', 1), ('b', 2), ('c', 3)]

Commit: test showing n_tuple conversion to list.

Reflection

Let’s settle down here and reflect on what we’ve done. This article is already quite long enough.

We are building up Extended Set Theory incrementally, learning about it and what it might be good for. As such, these kinds of things are almost inevitable:

  1. We’ll table questions and answer them later;
  2. We’ll occasionally make determinations that are not ideal;
  3. We’ll occasionally make mistakes that are not covered by our tests;
  4. We’ll occasionally get things right.

Today, we found a tabled item that led us first, to a related test that was running for the wrong reasons. We fixed it to work correctly. Then we found a few tests for the topic we were looking for and cleaned them up to be a bit more readable.

We recognized a legitimate question, whether an n-tuple should always iterate its elements in numeric order, waffled a bit, considered sorting, and decided to make that a rule. We wrote a test to confirm that rule.

The fact that an XSet can be converted to a Python list of tuples came to mind, and we wrote a test to document that fact.

This is what software development really is!

There are those who would tell us that before we do a software product, we should have it all figured out, all the questions asked and answered, and everything planned to the point where we can say what will be done, and when. I would like to be able to get some money down on the “don’t” with these people, because while it may have happened somewhere and sometime, this can only happen if there is no learning whatsoever taking place over the course of the entire effort. (Well, I suppose it could happen if all the learning was happy little surprises, but again, I want to get some money down on the “don’t”.)

In this effort, almost the entire purpose is learning, so that I am free to divert in any direction that strikes me. If this were an actual product effort, I’d have to be proceeding differently, delivering user- and management-visible progress toward a growing vision of what the product would be. I have tried that in the past, using Extended Set Theory, and not to good results.

Much of the fault was my own: I chose to drive the team to understand the theory before building anything useful. In the instances in hand, that was not viable: the powers that be ran out of patience before we had learned as much as I thought we needed to learn. My bad.

I did not know then what I know now, and had I known it, I would at least have been much more focused on delivering user-visible, user-valuable software. I am not sure that we could have met our two goals, to deliver something useful, and to deliver something set-theoretic. We would at least have had a chance to deliver something useful, and then perhaps a second-generation product with more set theory, and so on. We’ll never know, because we don’t get to re-run history.

To the best of my knowledge there does not exist a single running product based on extended set theory, anywhere in the world. (There might be a near-product: I know that the one we were working on at Comshare was useful enough that after the company was sold, some of the developers used the product at home. We were so close, though I had been driven out well before then.)

You might enjoy this article by Esther Schindler on the state of “Agile” these days. One key finding: management often wears agile garments but does not allow agile practices. And that basically does not work.

What works is a bit more like what you see here, continual learning and discovery.

Mind you, however, that this XST effort is not producing enough user-visible management-visible results for safety in a business environment. On the other hand if this were a real product effort, we would have more than one decades old man on the team and we would have business-side people collaborating to make sure we deliver enough goods to keep the money flowing.

Here, we can lean to the learning side more than we might if we were in this as a job. But the discovery, the learning, the mistakes and recoveries … those would always be there. They always are, even in a pre-planned-to-the-max effort. Why? Because we are humans.

Software development is a human activity, at least for now, and small steps are the way it works best.

At least that’s the view from here in the midst of it.

See you next time!