The Robot World Repo on GitHub
The Forth Repo on GitHub

I think yesterday’s experiment with a pure interpreter has told us enough to decide not to do it. We’ll discuss that, what it implies about experiments, and then look at next steps. Result: We have REPL!

Aside from pure interest and exercise, the point of yesterday’s experiment was to determine whether a pure interpreter approach to Forth would be better in some way, such that we might want to move in that direction. The experiment has succeeded, in my opinion, and the answer is “No”. I’ll provide some reasons below, but the real reason is that my very experienced intuition says “No”. With as much as we have in place, if this was a really tasty idea, it would be clear by now. We’d be seeing differences and thinking how much nicer the world would be if we had those differences. That’s not happening. I am confident that if this was a really good idea, I’d see it by now. But let’s suppose we needed “reasons”.

  • It will be slower. It looks up every word every time, and our existing scheme has a pointer to each called word in the word that calls it.
  • It has complexity in all the same places. It will handle IF-THEN and looping in ways that are similar to what we have now, and that will be no simpler, perhaps even more complicated.
  • It will be slower. It searches each conditional and loop every time, while our existing scheme has compiled them to internal operators and patched them so that they jump over code instead of trolling through it.
  • There is nothing particularly tasty going on in the code. There’s nothing that seems to be able to be done in a notably simpler, faster, or more elegant way than in our existing version.

In its favor? Well, it might be smaller code. But from the look of the interpreter, that smaller code is likely to have a much more conditional state-focused style.

We don’t really care much about smaller, though we do care about clarity. And we don’t really much care about speed: my computer is more than a thousand times faster than a classical Forth computer back in the day. (That is truly amazing. I wish humanity had become a thousand times better over the same interval.)

I am confident that by this time in an experiment, I have one of three feelings:

  1. This is sweet! Let’s see how we can get this good stuff!
  2. This is interesting. Let’s experiment a bit more.
  3. Meh.

On this one, the result is “Meh”. Experiment succeeds: we have our answer. We now return to our regularly scheduled programming.

What’s Next?

Good question. Let’s make a list.

Active Word?

There is a card saying “does active word work if there is a secondary inside a word needing to be parsed?” I do not understand that and would have to troll back through a few past articles to find out what I was thinking. I’ll try to write dates on my cards in the future. One alternative to trying to figure out what that means is to continue with what we’re doing, confident that if there is an issue it will turn up. Another is to review how active word works. I think we’ll probably just carry on, trying to be alert to active word issues.

REPL

Reader Rob—thanks, Rob!—points out that I need a REPL1. He is right. In fact, I have been running the py4fun Forth in a terminal window for a week or so now. I don’t do much with it, but I might. I do have this implemented:

Forth> 19 c
-7.222222222222222

The word c is defined to convert the top of the stack from Fahrenheit to Celsius, for my convenience in telling my friends from more enlightened lands how bloody cold it is here. Not a terribly advanced use, I grant, but you never know what I might do with it.

I’ve deferred having a REPL because our Forth is intended to run headless. It cannot issue a prompt to the robot programmer: it can only return results including error indications. That said, it would be very useful to a would-be robot programmer to be able to do as much work as possible in a Forth with a REPL, so that they can test.

Anyway, yes we need a REPL.

Rob was writing in response to my feeling rather blah about the Forth exercise, a few days back. He went on to point out that I would benefit from finding things to do with my Forth, and he is right again. A good reason to get a REPL in there.

That said, the py4fun Forth is really rather nice in its own way, and I could use and enhance it just as well as my own. That’s somehow not as tasty, using a program that I didn’t write. Not that I have written any of the other programs I use … anyway it seems wrong to have a Forth of my own and use py4fun’s.

Words from Files

It would be very useful to be able to put Forth into a text file and load it. It would be convenient for defining words like the ones I typed in to get c to work:

 : centigrade 32 - 5 * 9 / ;
 : c centigrade . ;

If py4fun Forth crashes, it will lose those definitions. I’d like to be able to put them on a file and load them in. I would also think it would be useful if my Forth could dump the dictionary words back out in colon-definition form.

File and Directory Operations.

Clearly we’d need at least some file and folder code to support loading, although that might be minimal. But to be a useful tool on the desktop we’d need more.

Home Automation?

I wonder whether there would be some way to rig up home automation stuff so that I could write Forth words to do things. I don’t have much home automation, just some lights, because my wife would kill me if I were to automate the doors or thermostat, and rightly so. But it might be fun, and my iPads and HomePods can do it, so there must be a way.

Weird Idea
I’m listening to the HomePod playing music right now. The Mac terminal program has a command to speak words. I wonder if I could get the Mac to tell Siri what to do … but I digress.

Plan

OK, REPL it is. You’ll find this curious or perhaps incredible, but I do not know how to package up a Python program so that it will run in the terminal or with a text window of our own invention. I’ll study that and continue from here …

Monday 0800

The above was written Saturday morning. Things intervened. The less said about that the better.

I think that I have never written a Python program that used terminal input. I have run some. Today, we need to correct that gap in my education. I’m told that we need a main program and one of those lines that triggers it.

if __name__ == '__main__':
    print("Hello World")

Woot! I get this:

/Users/ron/PycharmProjects/FORTH/.venv/bin/python /Users/ron/PycharmProjects/FORTH/source/main.py 
Hello World

Process finished with exit code 0

This seems promising, Deep research tells me that the built-in function input gets a line from the keyboard. We’ll try that.

if __name__ == '__main__':
    line = input()
    print(line)

With only a bit of difficulty getting the cursor into the right window:

/Users/ron/PycharmProjects/FORTH/.venv/bin/python /Users/ron/PycharmProjects/FORTH/source/main.py 
i am typing here
i am typing here

Process finished with exit code 0

OK, emboldened by these amazing successes, let’s go wild here.

from source.forth import Forth

if __name__ == '__main__':
    forth = Forth()
    line = input('F> ')
    forth.compile(line)

Note that my extensive research tells me that input will prompt with whatever parameter it is given. This is high-tech stuff here, folx!

And this, leaving out the initial long line from PyCharm:

F> 2 2 + .
4 
Process finished with exit code 0

So this is good. It’s time to actually think a bit about what’s needed. On the face of it, mostly what we need is to do this in a loop. We should allow for exiting the loop. The py4fun Forth lets you type ‘bye’ for that purpose. There is another issue that will arise, I think, having to do with input that is not complete, such as a colon definition that spans more than one line. And, of course, we’ll need to do something about errors.

All this in due time. We’ll munch forward, learning as we go.

if __name__ == '__main__':
    forth = Forth()
    while True:
        line = input('F> ')
        if line == 'bye':
            break
        forth.compile(line)

I test ‘bye’ first, and that works as advertised.

F> bye

Process finished with exit code 0

Now we’ll try a few statements and then a colon definition.

F> 3 1 + .
4 F> 2 2 * .
4 F> : c 32 - 5 * 9 / ;
F> 36 c
F> 36 c .
2 F> .
2 F> .
Traceback (most recent call last):
  ...
  File "/Users/ron/PycharmProjects/FORTH/source/stack.py", line 31, in pop
    return self.stack.pop()
           ^^^^^^^^^^^^^^^^
IndexError: pop from empty list

Process finished with exit code 1

So a few semi-surprises there. The output from my . operation seems not to include a return. That’s OK-ish. Forth generally just spaces after an output, and has a keyword to emit a return. Sure enough, here’s .:

        self.pw('.',    lambda f: print(f.stack.pop(), end=' '))

So that’s fine, although I think I’d rather have a return before the next ‘F>’. Should be no problem. The traceback isn’t so fun, but what if we just did the work in a try/except?

if __name__ == '__main__':
    forth = Forth()
    while True:
        line = input('F> ')
        if line == 'bye':
            break
        try:
            forth.compile(line)
        except Exception as e:
            print(e)

That works much as we might like.

F> 2 2 * .
4 F> 2
F> .
2 F> .
pop from empty list
F> : c 32 - 5 * 9 / . ;
F> 36 c
2 F> 5 0 /
integer division or modulo by zero
F>

We have some work to do to get the new lines where we want them, but that’s not important now. Still, it’s probably easy and I crave easy this morning.

The Forth convention is to type ‘ok’ at the end of a successful line. Let’s see …

if __name__ == '__main__':
    forth = Forth()
    while True:
        line = input('F> ')
        if line == 'bye':
            break
        try:
            forth.compile(line)
            print('ok')
        except Exception as e:
            print(e, ' ?')

And we get this result.

F> 2 2 + .
4 ok
F> 2 2 2 2 . . . .
2 2 2 2 ok
F> 5 0 /
integer division or modulo by zero  ?
F> : c 32 - 5 *
Unexpected end of input  ?
F>

Everything went as anticipated, so I tried an incomplete definition and got that final error. What we really want, stealing an idea from somewhere, maybe py4fun, is this:

F> : c 32 - 5 *
...>

The … prompt, whatever we choose it to be, will signify that Forth is still in the process of doing a definition or whatever.

Our Forth, as written, has only really been used under controlled conditions. Even when we’ve given it bad input, it has been in the context of a test. We haven’t addressed what it should do to be able to continue.

I try this, typing into the same prompt, which has not terminated yet:

F> : c 32 - 5 *
Unexpected end of input  ?
F> 9 / . ;
ok
F> 36 c
4 ok
F>

Here I typed part of the Celsius definition, then return, then the rest, then tested it. And it worked. So Forth was left in the middle of the definition, and just carried on. So let’s field that end of input error specially and then see what we get.

We have this test:

    def test_partial_definition(self):
        f = Forth()
        with pytest.raises(ValueError) as e:
            f.compile(': FOO 444 222 +')
        assert str(e.value) == 'Unexpected end of input'

So presumably somewhere I can find:

    def compile_a_word(self):
        self.word_list = []
        while True:
            token = self.next_token()
            if token is None:
                raise ValueError('Unexpected end of input')
            ...

I’m not sure about the Pythonic thing here. Arguably we should create our own Exception type here. Failing that, we should pick a sensible existing exception. Is this a value error? I guess in the sense that we expect a string from next_token and got None.

Python has EOFError and TypeError, among myriad others, that might apply here. I think in any case if we were to detect this particular error, we could get what we want.

if __name__ == '__main__':
    forth = Forth()
    prompt = 'Forth> '
    while True:
        line = input(prompt)
        if line == 'bye':
            break
        try:
            forth.compile(line)
            print('ok')
            prompt = 'Forth> '
        except Exception as e:
            if str(e) == 'Unexpected end of input':
                prompt = '...> '
            else:
                print(e, ' ?')

That’s rather messy but it works as intended:

Forth> 2 2 + .
4 ok
Forth> : c 32 - 5 *
...> 9 / ;
ok
Forth> 36 c .
4 ok
Forth>

I think we should commit this. Commit: Simple main provides simple REPL.

Reflection

So far so good. A quick test tells me that main.py will not run at a real Terminal prompt. Module not found error. I’ll have to learn how to package up a Python program. That’s a really odd thing not to know how to do, isn’t it? But there you have it: I’ve never done it because I’ve never needed it. So while you probably know exactly what I should do, and what I probably should have been doing right along, I get to look it up and learn it. So I am one of the lucky ones who get to learn this useful thing.

A quick change to put the file at the top of the hierarchy, i.e. not in the source subfolder, makes it work. that’s probably not how it should be done. I’ll do the research.

Be that as it may, and granting that the __main__ code is quite ad-hoc and not at all pretty, what we have here is a simple Forth REPL that actually works. Doubtless there are things to improve, but this is a lovely little step forward and we’ll congratulate ourselves just as if it had been the greatest discovery ever made.

Summary

Small wins. If we can only celebrate huge wins, we’ll have very little celebration. In these times, I know that I’m looking for any little taste of pleasure, joy, love, friendship. One of my joys is setting out to make the computer do something a bit more like what I want, and seeing that happen.

I hope that you can find ways to find little jolts of joy, and that if you program for work or pleasure, that you can find ways to make your programming provide a continuous flow of little bits of joy.

See you next time, when maybe I’ll know more about packaging. But it works!!


  1. Read-Eval-Print-Loop. A command-line prompt where we can type Forth and get results. ↩