Further Forth
There’s refactoring to be done, including, possibly, some Big Changes. What shall we do? How shall we proceed? (Results: Two small steps; much confusion.)
We have a refactoring theme on hold, the changes that put special functions inside the lexicon definitions, rather than as methods in the Forth class. Related to that is the notion that the lexicon should be a class of its own, which would make it easier and more sensible to move the built-in definitions out of the Forth class.
And but also, yesterday I managed to change one of my very internal words, *DO, from being implemented in Python code, to being compiled as a word. It would be nice, in some sense, to do more words that way. My reading and thinking makes me believe that for many of the candidates, we’d need words that allow other words to build up definitions, access separate words in the current (secondary) word, and so on.
Real Forth systems manage that a bit more easily, because their word definitions are basically just a long list of cells, and there are special words that return useful things such as the address in the list of the current word or next word. The thing is, Forth as it stands is a completed thing. Every implementation does things a bit differently, but the good reference systems are “done”, polished, and the thinking behind their specialized word set is not apparent. At least not to me. So I’m not sure just what to do, what special words to implement to let us build up more words in Forth and fewer in Python.
Now one very good question would be “Why would you care about doing more words in Forth itself?”
Possible answers include:
- It’s interesting and sort of fun;
- With the right words defined, users could build more compiling words and such;
- We might more readily borrow Forth code from somewhere else;
But we have no users and probably never will. It is barely possible that I would push this Forth far enough to make it a useful utility for me on my Mac. It would be “nice” if I could build new words as needed, without resorting to rebuilding the system in Python.
You see why it’s a good question: it comes down to me figuring out what will be interesting and fun. And, ideally, with some hope of entertaining some readers and, just possibly, inspiring a few to find interesting, fun, and even valuable things for themselves.
Let’s get down to cases. We still have at least one Forth word that requires a method in the Forth class. (The rest, except for yesterday’s *DO, are also in Python, but are in lambdas and do not add methods to the Forth class.)
In fact there are three:
def i_word(self):
index = self.return_stack[-1]
self.stack.push(index)
def star_loop(self):
beginning_of_do_loop = self.next_word()
index = self.return_stack.pop()
limit = self.return_stack.pop()
index += 1
if index < limit:
self.return_stack.push(limit)
self.return_stack.push(index)
self.active_word.skip(beginning_of_do_loop)
def zero_branch(self):
branch_distance = self.next_word()
if self.stack.pop() == 0:
self.active_word.skip(branch_distance)
I know how to do the i_word. There can be another word like >R and R>, namely R@. R@ pushes the value on top of the R stack to the regular stack. So we can define that:
lex.append(PrimaryWord('R@', lambda f: f.stack.push(f.return_stack.top())))
And I implemented stack.top as well:
class Stack:
def top(self):
return self[-1]
It seemed like the logical thing to do. Of course it is one method call deeper but still, it’s how we do things.
Are you wondering how that works? Stack already has this:
class Stack:
def __getitem__(self, item):
return self.stack[item]
And now:
def define_skippers(self,lex):
lex.append(PrimaryWord('*#', lambda f: f.stack.push(f.next_word())))
lex.append(PrimaryWord('*IF', lambda f: f.zero_branch()))
lex.append(PrimaryWord('*ELSE', lambda f: f.active_word.skip(f.next_word())))
lex.append(PrimaryWord('*UNTIL', lambda f: f.zero_branch()))
lex.append(PrimaryWord('*LOOP', lambda f: f.star_loop()))
self.compile(': *DO SWAP >R >R ;')
self.compile(': I R@ ;')
And we are green, and yes we do have a test for that:
def test_initial_do(self):
f = Forth()
s = ' 5 0 DO I 10 * LOOP '
f.compile(s)
assert f.stack.stack == [0, 10, 20, 30, 40]
Commit: implement I word as secondary just as a flex.
I freely grant that I do not know just now how to implement J and K words, which have to dig deeper into the stack than that. We probably need some words for the return stack corresponding to the regular words like DUP and SWAP.
OK. We were really here to look at the remaining star word:
def star_loop(self):
beginning_of_do_loop = self.next_word()
index = self.return_stack.pop()
limit = self.return_stack.pop()
index += 1
if index < limit:
self.return_stack.push(limit)
self.return_stack.push(index)
self.active_word.skip(beginning_of_do_loop)
We can imagine part of this in Forth:
R> R> SWAP 1 +
That would give us [limit, index+1] on the stack. We need to know whether index < limit, so we could use a word we don’t have but could have, 2DUP, to duplicate both and do the compare:
R> R> SWAP 1 + 2DUP < IF SWAP >R >R JUMP_BACK ELSE DROP DROP SKIP THEN
We have two words there that no one has ever heard of before, JUMP_BACK, and SKIP.
The *LOOP is compiled like this:
def _loop(forth):
key, jump_loc = forth.compile_stack.pop()
loop = forth.find_word('*LOOP')
forth.word_list.append(loop)
forth.word_list.append(jump_loc - len(forth.word_list))
The key fact is that following *LOOP is a word containing the distance in words, to jump back to the front of the loop. So if we had a word to pick that up and do the skipping, and another word to just consume it so that we don’t try to execute it. We could probably implement *LOOP in Forth, as shown above.
All that assumes that the Forth code there is correct.
We could write a test:
def test_compiled_star_loop(self):
f = Forth()
star_loop = ': *LOOP R> R> SWAP 1 + 2DUP < IF SWAP >R >R JUMP_BACK ELSE DROP DROP SKIP THEN ;'
f.compile(star_loop)
print("this is the one")
s = ' 5 0 DO I 10 * LOOP '
f.compile(s)
assert f.stack.stack == [0, 10, 20, 30, 40]
With the f.compile(star_do) commented out, the test runs. Left in we get:
> raise SyntaxError(f'Syntax error: "{token}" unrecognized')
E SyntaxError: Syntax error: "2DUP" unrecognized
We’ll need to implement that. And, unfortunately, I don’t see how to do it as a lambda.
def define_stack_ops(lex):
def _2dup(forth):
top = forth.stack[-1]
bot = forth.stack[-2]
forth.stack.push(bot)
forth.stack.push(top)
lex.append(PrimaryWord('2DUP', _2dup))
And yes, I do have a test:
def test_2DUP(self):
f = Forth()
s = '1 5 2DUP'
f.compile(s)
assert f.stack.stack == [1, 5, 1, 5]
I suppose our *LOOP test is failing on JUMP_BACK now?
> raise SyntaxError(f'Syntax error: "{token}" unrecognized')
E SyntaxError: Syntax error: "JUMP_BACK" unrecognized
Reflection
I’m just kind of fumbling forward here, to see how far we can go with this. I have only a vague notion of how to do the JUMP_BACK, but we can certainly look at the original *LOOP to get some sense of it.
def star_loop(self):
beginning_of_do_loop = self.next_word()
index = self.return_stack.pop()
limit = self.return_stack.pop()
index += 1
if index < limit:
self.return_stack.push(limit)
self.return_stack.push(index)
self.active_word.skip(beginning_of_do_loop)
The problem is that self.next_word() looks at the word beyond the current word in the definition we are running. If the word in question, *LOOP, is implemented as a secondary word, then it will not have access to that value … or, if it will, I can’t see right now how we could find it.
[Further fumbling happens here. Not worth the space.]
I’ll have to come at this with fresh eyes. My current eyes and brain are dull.
Summary
We have made a little progress. We implemented two useful primary words, R@ and 2DUP, and replaced a method in the Forth class with a compiled definition for the I word. That one we get extra credit for, because it’s only the second built-in word we have managed to define that way. And we have a possibly useful test written to let us test a secondary word for *LOOP if we can figure out some details.
I’m not sure about those details. I note that in the py4fun Forth, they have a special ability for a word to return, instead of None, an integer that is used to skip forward or backward as needed. We might have to steal that idea, but their implementation is quite different from ours. Further study and experimentation is called for.
I’ll browse further in Loeliger and forth.com. And I definitely need some better tools for inspecting the compiled Forth words.
It’s always something, I can’t say I’m delighted with progress this morning, but we have a couple of small accomplishments, and a lot more experience.
Good judgment comes from experience.
Experience comes from bad judgment.
I’m not quite sure why that came to mind.
See you next time!