Forth: Mistake?
I’ve made some improvements. And I have a concern. Longish article, probably a quick read.
Concern
Our initial predefined basic lexicon is currently coded like this:
class Forth:
def __init__(self):
self.stack = Stack()
self.lexicon = []
self.define_primaries()
def define_primaries(self):
lex = self.lexicon
stack = self.stack
lex.append(PrimaryWord('DROP', stack.pop))
lex.append(PrimaryWord('DUP', stack.dup))
lex.append(PrimaryWord('OVER', stack.over))
lex.append(PrimaryWord('ROT', stack.rot))
lex.append(PrimaryWord('SWAP', stack.swap))
lex.append(PrimaryWord('+', lambda: stack.push(stack.pop() + stack.pop())))
lex.append(PrimaryWord('-', lambda: stack.push(stack.swap_pop() - stack.pop())))
lex.append(PrimaryWord('*', lambda: stack.push(stack.pop() * stack.pop())))
lex.append(PrimaryWord('/', lambda: stack.push(stack.swap_pop() / stack.pop())))
lex.append(PrimaryWord('SQRT', lambda: stack.push(math.sqrt(stack.pop()))))
As a preview, you might notice that some of our primaries are now sending messages to stack. Yes, I’ve built a small Stack class. We’ll look at it soon. But the concern is this: those Words all refer to stack, which is set to self.stack in the define_primaries method. So they are bound to the current instance of Stack that is stored in self.stack. If that attribute is ever changed to contain a new Stack, I think things will break.
Let’s verify that concern with a test.
def test_changing_stack(self):
f = Forth()
f.stack.extend([3, 4])
f.find_word('+').do(f)
assert f.stack.pop() == 7
This passes. This should not:
def test_changing_stack(self):
f = Forth()
f.stack = Stack() # Never touch me there!
f.stack.extend([3, 4])
f.find_word('+').do(f)
assert f.stack.pop() == 7
And it does:
def pop(self):
> return self.stack.pop()
E IndexError: pop from empty list
We could, of course just protect the stack, with a property or something. And no one should be fiddling with the stack anyway. So we can let this go. But I think we need to do something different anyway. It turns out, which I didn’t exactly recall until I read the code again, we do pass the Forth instance to every Word on do:
class SecondaryWord:
# why don't we just store the word in the lexicon? it's no larger than the index.
def __init__(self, name, word_indices):
self.name = name
self.word_indices = word_indices
def do(self, forth):
lexicon = forth.lexicon
for word_index in self.word_indices:
lexicon[word_index].do(forth) # <=====
So let’s use it in our PrimaryWords, which will fix this broken test with any luck at all. We’ll change this:
class PrimaryWord:
def do(self, _forth):
self.code()
To this:
class PrimaryWord:
def do(self, forth):
self.code(forth)
Naturally tests break. We’ll look at one to see what’s up.
def do(self, forth):
> self.code(forth)
E TypeError: Forth.define_primaries.<locals>.<lambda>() takes 0 positional arguments but 1 was given
As expected. Perfect. Now fix all the primaries:
def define_primaries(self):
lex = self.lexicon
stack = self.stack
lex.append(PrimaryWord('DROP', stack.pop))
lex.append(PrimaryWord('DUP', stack.dup))
lex.append(PrimaryWord('OVER', stack.over))
lex.append(PrimaryWord('ROT', stack.rot))
lex.append(PrimaryWord('SWAP', stack.swap))
lex.append(PrimaryWord('+', lambda forth: forth.stack.push(forth.stack.pop() + forth.stack.pop())))
lex.append(PrimaryWord('-', lambda forth: forth.stack.push(forth.stack.swap_pop() - forth.stack.pop())))
lex.append(PrimaryWord('*', lambda forth: forth.stack.push(forth.stack.pop() * forth.stack.pop())))
lex.append(PrimaryWord('/', lambda forth: forth.stack.push(forth.stack.swap_pop() / forth.stack.pop())))
lex.append(PrimaryWord('SQRT', lambda forth: forth.stack.push(math.sqrt(forth.stack.pop()))))
That fixes 3 of 7 broken tests. We also begin to see why Guido doesn’t like lambdas: those are getting messy. We’ll address that.
I imagine the other four fails relate to the first five Primaries, which are bound improperly to the stack temp. Yes, such as:
def do(self, forth):
> self.code(forth)
E TypeError: Stack.rot() takes 1 positional argument but 2 were given
I think we have to make those other Primaries into lambdas. And we should remove the stack temp.
def define_primaries(self):
lex = self.lexicon
lex.append(PrimaryWord('DROP', lambda f: f.stack.pop()))
lex.append(PrimaryWord('DUP', lambda f: f.stack.dup()))
lex.append(PrimaryWord('OVER', lambda f: f.stack.over()))
lex.append(PrimaryWord('ROT', lambda f: f.stack.rot()))
lex.append(PrimaryWord('SWAP', lambda f: f.stack.swap()))
lex.append(PrimaryWord('+', lambda f: f.stack.push(f.stack.pop() + f.stack.pop())))
lex.append(PrimaryWord('-', lambda f: f.stack.push(f.stack.swap_pop() - f.stack.pop())))
lex.append(PrimaryWord('*', lambda f: f.stack.push(f.stack.pop() * f.stack.pop())))
lex.append(PrimaryWord('/', lambda f: f.stack.push(f.stack.swap_pop() / f.stack.pop())))
lex.append(PrimaryWord('SQRT', lambda f: f.stack.push(math.sqrt(f.stack.pop()))))
We are green. Even the test that slams the stack works, although it’s still a very bad idea do to that.
Commit: fix bogus reference to stack in define_primaries.
Renaming the forth parameter to f has reduced my tension over the complexity of those lambdas. We could perhaps be more “Pythonic” by making them all methods of Forth or something, but I think the extra code would obfuscate what’s going on. We’ll let this ride until it bothers us more. That day may never come.
Changes
While you were away, or while I was away, depending on how you look at it, I did a little work. I wrote a Stack class and used it. Take a quick look at this:
class Stack:
def __init__(self):
self.stack = []
def dup(self):
self.push(self.stack[-1])
def extend(self, items):
self.stack.extend(items)
def over(self):
self.push(self.stack[-2])
def pop(self):
return self.stack.pop()
def push(self, item):
self.stack.append(item)
def rot(self):
stack = self.stack
top, middle, bottom = stack.pop(), stack.pop(), stack.pop()
self.extend([middle, top, bottom])
def swap(self):
stack = self.stack
top, under = stack.pop(), stack.pop()
self.extend([top, under])
def swap_pop(self):
# (_ under top -> _ top ) -> under
self.swap()
return self.pop()
def __eq__(self, other):
return self.stack == other
As you can see, I implemented most of the Forth stack-adjusting words directly on my Stack object, which, as we saw, simplifies a number of the lambdas we have in the PrimaryWord instances. The most interesting of these, well, it’s a tie between rot, which you have to think about a lot when you code it, but not when you use it, and swap_pop, which is useful for getting around that hack of negating or dividing into 1 that we had in primaries - and /.
lex.append(PrimaryWord('-',
lambda f: f.stack.push(f.stack.swap_pop() - f.stack.pop())))
lex.append(PrimaryWord('/',
lambda f: f.stack.push(f.stack.swap_pop() / f.stack.pop())))
Those are currently the only two uses of swap_pop. I wonder if Forth has an operator that does that. We could provide it if it has a name.
A search of the Internet doesn’t find a useful answer but does turn up some really odd operators including NIP, TUCK, PICK, and ROLL. We’ll ignore those entirely. One of the articles up and says if you are using these, you’re probably doing it wrong.
Anyway, I think adding the Stack class and using it was all I really changed while you were away.
I think this article is long enough. Let’s sum up and then start another.
Summary
We observed a concern with the references to the stack, which were working via aa rather bizarre closure effect, and resolved that by using the already-existing parameter to the do, passing it down to our primitives.
Then we just took a quick look at the longish but trivial Stack class. Coming right up, some real implementation. See you there!