The Repo on GitHub

This time it’s going to work, I’m sure of it. I promise not to change the title, or the blurb up to the next period if it doesn’t. (Results: As predicted! Out of the park! Yay, team!)

Probably I should have reviewed the code before calling that shot, but I am confident. Here, before reviewing the code, is my cunning plan.

The CREATE has to work with or without a DOES>. CREATE defines a word that pushes the address of the next available heap cell onto the stack. The word’s name is the token following CREATE in the input.

So we’ll have CREATE actually define that simple word. In essence we have no choice. So we will cause the Lexicon to remember the word most recently created. And we will cause DOES> to fetch that word and extend it with the code that follows it.

Currently, I believe that DOES> tells the current active word to quit. Instead, we’ll have a method on SecondaryWord that will copy its remaining contents to another word.

I’m not sure quite where all the various bits of logic will wind up. We may need some refactoring after this all works: we usually do. But I’m reasonably confident in the plan.

Larger-Scale Observation

This “feature” has been a real bugaboo for me. I had two terrible days in a row of trying to work it out, with the final result of each day being “Bah!”. Then after reading a lot of Forth info, and after a brief but very valuable exchange with GeePaw Hill, I made some small but important progress yesterday.

And I really am quite confident that we’ll get this done, probably today. If I’d just get around to it. Patience, dear, we’ll start shortly. I want to make this point:

Sometimes the bear bites us. When it does, thinking and study are of value. But of even more value are the other members of our team or our network. Sometimes, as with Hill, they’ll have a useful bit of clarity to offer. Other times, they’ll offer random ideas that will help jar loose useful ideas of our own. Sometimes they’ll ask questions that help clarify our thinking. Sometimes it’s just good to have someone to kick it around with.

In these times of remote work and other forms of isolation, our connections may be on line rather than in person. I think that’s tragic, though it is a price I am personally willing to pay to avoid some rando bringing me a disease. But on line human contact, especially when we can use Zoom or a similar thing, is good for the soul, and even if you don’t have a soul, it can be good for the code.

Something to think about.

CREATE

Currently, these are CREATE and DOES>

    def _create(forth):
        name = forth.next_token()
        forth.compile_stack.push(name)

        def _does(forth):
            name = forth.compile_stack.pop()
            forth.active_word.finish()

Let’s begin with a new test for CREATE.

    def test_create_alone(self):
        f = Forth()
        f.compile('CREATE FOO')
        f. compile('FOO')
        assert f.stack.pop() == 0
        f.compile('3 FOO !')
        assert f.stack.is_empty()
        f.compile('FOO @')
        assert f.stack.pop() == 3
        assert f.stack.is_empty()

I kind of went hog wild with that test. Could have done a simpler one. In fact, let’s do. I’ll comment out all but the first three lines.

You might wonder why I’d do that. I’m doing that because it will allow smaller steps to pass pieces of the test, and that will be good. In fact, I’m going to delete those lines entirely, to allow for a different path if I happen to want one.

    def test_create_makes_a_word(self):
        f = Forth()
        f.compile('CREATE FOO')
        f. compile('FOO')

That should be pretty easy to pass. I like that in a test.

        def _create(forth):
            name = forth.next_token()
            word = SecondaryWord(name, list())
            forth.lexicon.append(word)

        def _does(forth):
            forth.active_word.finish()

The test passes. Ha! That’s why I made the test so tiny. Commit: working on create-does>.

Wait, you say, that doesn’t even work! True, but all the tests pass, therefore nothing important is using it yet. That’s the marvel of tiny steps: we can keep all our work in HEAD, nice and fresh.

OK, in the spirit of tiny steps, here comes Fake It Till You Make It.

    def test_first_create_is_at_zero(self):
        f = Forth()
        f.compile('CREATE FOO')
        f. compile('FOO')
        assert f.stack.pop() == 0

Now this is supposed to be the next available heap location. But we can fake it, just returning zero:

    def _create(forth):
        name = forth.next_token()
        lit = forth.find_word('*#')
        word = SecondaryWord(name, [lit, 0])
        forth.lexicon.append(word)

Green. Commit same message.

Wait, you say, that still doesn’t even work! See above. All the tests pass: I can commit. Why do I commit? Why is this reasonable? It’s reasonable because I’ve driven out a bit of the necessary code, the creation of an actual list that we can build upon. And I can commit because it works. It’s just not very complete.

Question
We know that we’re going to edit this word list with DOES>. Are we safe passing in this list? Will we get a new one every time? It would be bad if there was only one and two creates doing different things stepped on each other.

I’m going to belay that question for now, but I put a sticky note on my desk to check it. I should probably just know what Python does with this, but I am not certain and I do know that default arguments always provide the same object, so I have the concern.

We need a harder test.

    def test_second_create_is_at_one(self):
        f = Forth()
        f.compile('CREATE FOO')
        f.compile('CREATE BAR')
        f. compile('BAR')
        assert f.stack.pop() == 1

Fails, no big surprise there. Let’s get the actual heap work done. No, wait, there’s a better test.

    def test_create_is_at_heap_end(self):
        f = Forth()
        f.compile('VARIABLE FOO 3 ALLOT')
        f.compile('CREATE BAR')
        f. compile('BAR')
        assert f.stack.pop() == 3

Ah. I almost made a mistake. I ran this test first:

    def test_create_is_at_heap_end(self):
        f = Forth()
        f.compile('VARIABLE FOO 3 ALLOT')
        assert len(f.heap) == 3

And it failed, because I was priming the heap with a zero cell. I’ve changed it:

class Forth:
    def __init__(self):
        ...
        self.heap = []
        ...

The heap now has an interesting property: it contains no data that has not been allotted. Therefore this test should pass:

    def test_variable_without_allot_fails(self):
        f = Forth()
        f.compile('VARIABLE FOO')
        with pytest.raises(IndexError):
            f.compile('FOO @')

OK, working as advertised. Green. Commit again, then back to work. changed heap to have only explicitly allotted words.

OK, this is of course passing:

    def test_create_is_at_heap_end(self):
        f = Forth()
        f.compile('VARIABLE FOO 3 ALLOT')
        assert len(f.heap) == 3
        # f.compile('CREATE BAR')
        # f. compile('BAR')
        # assert f.stack.pop() == 3

So we uncomment the last three to get the fail that drives out our next bit of code:

And our change is to get the current heap length and use that in the definition of the CREATE’s word:

        def _create(forth):
            name = forth.next_token()
            lit = forth.find_word('*#')
            word = SecondaryWord(name, [lit, len(forth.heap)])
            forth.lexicon.append(word)

Green. Commit: CREATE addresses next word in heap.

The Forth standard tells me: “CREATE does not allocate data space in name’s data field.”

So let’s see … do we have comma implemented? Not yet. OK, but we can do this for a test:

    def test_demo_create(self):
        f = Forth()
        f.compile('VARIABLE FOO 3 ALLOT')
        assert len(f.heap) == 3
        f.compile('CREATE BAR 1 ALLOT')
        f. compile('BAR @')
        assert f.stack.pop() == 0
        f.compile('42 BAR !')
        assert f.stack.stack == []
        f. compile('BAR @')
        assert f.stack.pop() == 42

This is passing. Commit: CREATE complete but not pretty

It is coming up on time to do DOES>, but I think first we’d best make comma work, because our DOES> test wants to use it.

Best write a test for it.

    def test_comma(self):
        f = Forth()
        assert len(f.heap) == 0
        f.compile('33 ,')
        assert len(f.heap) == 1
        assert f.heap[0] == 33

And this should be easy:

    def _comma(forth):
        forth.heap.append(forth.stack.pop())

Green. Commit: Comma implemented.

That suggests another test that should pass:

    def test_create_comma(self):
        f = Forth()
        f.compile('CREATE BAR 19 , BAR @  23 +')
        assert f.stack.pop() == 42

Green. Commit: nifty test for CREATE and COMMA.

Reflection

We’ve just been screaming along here. We have seven commits in the last 50 minutes. That’s like seven minutes between commits, brilliant results when you consider all the thinking and typing and article writing that has gone on. We have almost 10,000 characters in this article right now.

I think things are somewhat robust, but probably not in quite the right places. In particular, it is starting to look like the heap should be some kind of an object, if only an instance of our Stack, class. We’re doing a lot of len and stuff with it.

But I do think that CREATE and comma are doing the right thing, if not quite with the best code we’re capable of. Make it work, then make it right, that’s the motto.

I think that all that remains is to complete our partial test for defining CONSTANT with CREATE, and make it work by making DOES> copy the code over.

I am going to take a break, however, and then come back to this. I want to cool down a bit. Don’t go away, I’ll be back on the next line. Or, if you like, take a break. I’m not your boss.

After a short interval …

OK, I’ve made my iced chai latte, and consumed much of it while finishing reading my current thriller. It’s time to finish up DOES>. Let’s see what we have for a test. We have a start at one, which is running green, telling me that it is not yet quite difficult enough:

    def test_compile_create_does(self):
        f = Forth()
        s = ': CONSTANT CREATE , DOES> @ ;'
        f.compile(s)
        word = f.find_word('CONSTANT')
        print(word)
        f.compile('11 22 33 2025 CONSTANT YEAR')

At this point, I think we’d find that YEAR has address 9 in it. Let’s move that long using some commas, just to make it a bit harder. This modified test will compile but fail with 3, not 2025:

    def test_compile_create_does(self):
        f = Forth()
        f.compile('1 , 2 , 3 ,')
        s = ': CONSTANT CREATE , DOES> @ ;'
        f.compile(s)
        f.compile('2025 CONSTANT YEAR')
        f. compile('YEAR')
        assert f.stack.pop() == 2025

And it does. Why? Because as things are now, YEAR is defined as [*#, 3], because 3 is the assigned heap location. We can check that the 2025 is stored there. Let’s do that.

    def test_compile_create_does(self):
        f = Forth()
        f.compile('1 , 2 , 3 ,')
        s = ': CONSTANT CREATE , DOES> @ ;'
        f.compile(s)
        f.compile('2025 CONSTANT YEAR')
        assert f.heap[-1] == 2025
        f. compile('YEAR')
        assert f.stack.pop() == 2025

The heap check passes, and the YEAR still fails with 3, not 2025. Our mission is for DOES> to copy everything after the `DOES> into the most recent word. We’d do it like this:

    def _does(forth):
        forth.active_word.copy_to_latest(forth.lexicon)

We don’t have such a method and the test tells us so:

    def _does(forth):
>       forth.active_word.copy_to_latest(forth.lexicon)
E       AttributeError: 'SecondaryWord' object has no attribute 'copy_to_latest'

Create the method:

    def copy_to_latest(self, lexicon):
        latest = lexicon.latest_word()
        while self.pc < len(self.words):
            w = self.next_word()
            latest.append(w)

This is sheer wishful thinking. Lexicon doesn’t know latest_word and SecondaryWord doesn’t know append. We will remedy that now.

class Lexicon:
    def __init__(self):
        self.lexicon = []
        self._latest = None

    def append(self, word):
        self._latest = word
        self.lexicon.append(word)

    def latest_word(self):
        return self._latest

We should be failing now on SecondaryWord.append.

    def copy_to_latest(self, lexicon):
        latest = lexicon.latest_word()
        while self.pc < len(self.words):
            w = self.next_word()
>           latest.append(w)
E           AttributeError: 'SecondaryWord' object has no attribute 'append'

That, too, should be easy to remedy:

    def append(self, word):
        self.words.append(word)

And our test is green, do you hear me, GREEN! Muhahahahahaha GREEN!!!

One more confirming test before I commit, not because I’m uncertain but because I want to watch how wonderful this is.

    def test_compile_two_constants(self):
        f = Forth()
        f.compile('1 , 2 , 3 ,')
        s = ': CONSTANT CREATE , DOES> @ ;'
        f.compile(s)
        f.compile('2025 CONSTANT YEAR')
        f.compile('1939 CONSTANT BIRTH_YEAR')
        f. compile('YEAR BIRTH_YEAR - 1 -')
        assert f.stack.pop() == 85

Passes! Why the extra -1? Because my birthday isn’t until the very end of the year and I’m not 86 … yet!

Commit: CREATE-DOES> implemented and passing all tests.

Summary

So, there you have it: ten tiny commits, straightforward progress, no surprises, mostly because our tests were tiny, so our incremental steps were tiny, and we never had to think very hard.

Was it easy? Well, today it was easy. Three days ago it was difficult: too difficult. Two days ago: too difficult. Yesterday, a good start, based on much clearer understanding of what had to be done.

Yesterday, I made my final test pass by making it simpler: the code just had to compile and execute without an exception. I kludged the DOES> to skip over the words that follow it, without executing them. That left us with just a few tasks for today, changing CREATE to make a complete word, implementing , to allocate and store into the heap, and making DOES> copy from the word containing the DOES> to the latest word in the lexicon.

I should mention why that is OK: The Forth standard actually says that DOES> is supposed to make changes to the most recent definition. This is definitely a hack, but it’s a hack that’s in the standard. There is a bit more to look into: the standard says “replace” the semantics of the most recent word, and what we have done is extend the most recent word’s semantics. I’ve made a note to look into that: the standard isn’t easy to understand.

That aside, I still want to ensure that we get a new list on each word we create, so that note endures on my keyboard tray as well.

And we surely want a smarter heap object and probably other refactoring.

But it works as intended, with the assistance of GeePaw Hill, Bill Wake, the Forth standards committee, R. G. Loeliger, Chris Meyers and Fred Obermann of py4fun, and Rational Matter of Juno Python, and surely a host of others I have forgotten.

I am claiming the win for the team. See you next time!