The Robot World Repo on GitHub
The Forth Repo on GitHub

#StopTheCoup! We need a FileProvider to feed us tokens from files. And we need file names, I suppose …

It’s odd, but I have only rarely had occasion to read or write files in Python. I have done it, perhaps in another world, but not often. So there will be some odd explorations here, as I work out how to do things that you, if you were here, could probably just go ahead and do. I almost miss you.

One issue with files delivering tokens to Forth is that malformed input of any kind will cause Forth to call abend and go to its prompt (at least until we figure out some other possibility). So if we are three files deep in INCLUDE (you better not do that but I suppose it has to work) there would be the issue of open files. We’ll have a stack, I suppose, but I think I’d prefer to keep them closed rather than open.

Python has this nice with capability for an open file, that is guaranteed to get the file closed if something goes wrong. So I’‘d prefer to use that. And I have a tentative plan:

Our FileProvider will open its file and read some large-ish number of lines, closing the file via with when it has the lines. If those ever get consumed, it will go back and get more. That will keep the file closed most of the time and it will mean that whatever stack of active files we have, we won’t need to do more than empty it. I like that better. YMMV.

I think I’ll start with some work in a test file for the FileProvider. We have a file test_providers, which will do just fine.

I’ll create a little include file at the top of the project.

: c 32 - 5 * 9 / ;
0 c 32 c -40 c . . .

That defines a word ‘C’ that computes the Celsius temperature given Fahrenheit. It doesn’t round, but this isn’t science here. After defining the word it computes 0F 32F and -40F and prints the three results. At main, that looks like this:

Forth>: c 32 - 5 * 9 / ;
Forth>0 c 32 c -40 c . . .
-40 0 -18 
Forth>

I note that I’m not getting ‘ok’ printed out. I thought we had left that working but I guess not. We’ll not divert to work on that but that area does need sorting.

I commit the file. I should have called it celsius, I suppose, but in my day we called it centigrade.

Now I can see about opening it and reading it.

    def test_open_file(self):
        with open('centigrade.forth', 'r') as file:
            pass

Woot! That runs (and fails if I give it a mistaken name). Let’s read from it.

    def test_read_file(self):
        with open('centigrade.forth', 'r') as file:
            lines = file.readlines()
        assert lines[0] == ': c 32 - 5 * 9 / ;\n'
        assert lines[1] == '0 c 32 c -40 c . . .'

Ha! That was a good reminder. The lines include the newline at the end, and I suppose on archaic Windows machines, there might be a carriage return as well. An there is no newline on the last line, because I didn’t type one.

I think our provider cascade will be trimming the lines anyway, but we should keep that in mind.

A bit of thinking about how to read some lines and not all of them leads me to think that we’ll just read in the file and be done with it. We’re not on a 4096-byte computer here, we’re on a 17,179,869,184 byte computer. We can afford to read the whole file in.

OK, let’s think about the FileProvider. Providers just have two methods, has_tokens, signifying that they still have something to provide, and next_token to provide a token. They typically use a StringProvider to provide the actual tokens. If by “typically” you mean “we have done it one other time”.

I sort of wish I had built a file that is easier to test but let’s see what happens. Here’s how we’ll use a FileProvider, at first guess:

    def test_file_provider(self):
        fp = FileProvider('centigrade.forth')
        tokens = []
        while fp.has_tokens():
            tokens.append(fp.next_token())
        assert tokens[0] == ':'
        assert tokens[2] == '32'
        assert tokens[-1] == '.'
        assert tokens[-5] == '-40'

That should drive out some code.

I nearly just typed this in. Truth be told it took a couple of tries:

class FileProvider:
    def __init__(self, filename):
        with open(filename, 'r') as f:
            self.lines = f.readlines()
        self.line_number = 0
        self.reader = StringProvider()

    def has_tokens(self):
        return self.reader.has_tokens() or self.line_number < len(self.lines)

    def next_token(self):
        if not self.reader.has_tokens() and self.line_number < len(self.lines):
            self.reader = StringProvider(self.lines[self.line_number])
            self.line_number += 1
        return self.reader.next_token()

Let’s remove that duplication:

class FileProvider:
    def __init__(self, filename):
        with open(filename, 'r') as f:
            self.lines = f.readlines()
        self.line_number = 0
        self.reader = StringProvider()

    def has_tokens(self):
        return self.reader.has_tokens() or self._has_lines_left()

    def _has_lines_left(self):
        return self.line_number < len(self.lines)

    def next_token(self):
        if not self.reader.has_tokens() and self._has_lines_left():
            self.reader = StringProvider(self.lines[self.line_number])
            self.line_number += 1
        return self.reader.next_token()

That went nicely.

It seems likely that I can put definitions in a file and initialize using them. Let’s rename the centigrade file to ‘init.forth’ in aid of that. PyCharm politely renames the file in the tests. Gotta love it.

Time to commit: Initial FileProvider working. Why did I say “initial”? Well, it’s early days and I’m not quite convinced that we’re done there, though we may well be.

I just looked up the standard for Forth word INCLUDE. It reads the file name from the input as a space-delimited token. We don’t need strings to do INCLUDE. Woot! Let’s write a test.

    def test_include(self):
        f = Forth()
        result = f.compile('include init.forth')
        assert result == 'ok'
        f.compile('32 c')
        assert f.stack == [0]

This should work, with the current init file. I think we’ll want to make a separate init from the file used in the tests. init might come in handy,

We’re getting this message instead of ‘ok’:

Expected :'ok'
Actual   :'Syntax error: "INCLUDE" unrecognized ?'

Let’s implement INCLUDE.

    def define_include(self):
        def _include(f):
            file_name = f.next_token()
            fp = FileProvider(file_name)
            f.main_loop(fp)

        self.pw('INCLUDE', _include)

I am nearly surprised that the test runs, but it does. Commit: initial implementation of INCLUDE.

I have to try this in main, it’s too good to pass up.

After some messing about I have improved the test and found out that the working folder in my main isn’t what I thought it was. The following works:

    def test_include(self):
        f = Forth()
        result = f.compile('include init.forth')
        assert result == 'ok'
        word = f.find_word('C')
        print(f'C is {word}')
        f.compile('32 c')
        assert f.stack.pop() == 0
        f.compile('-40 c')
        assert f.stack.pop() == -40
        print(f.stack.stack)
        assert f.stack.is_empty()

And in main:

/Users/ron/PycharmProjects/FORTH/source
ok
Forth>32 c
Syntax error: "C" unrecognized ?
Syntax error: "C" unrecognized ?
Forth>include ../init.forth
-40 0 -18 ok
ok
Forth>32 c
ok
Forth>.
0 ok
Forth>

That first line is me printing os.getcwd() which told me I needed the .. on the file name.

So INCLUDE is actually working. And I am tired. We’ll commit this, since we are green and them’s the rules. Commit: INCLUDE is working. needs to deal with missing files etc.

Let’s sum up.

Summary

The past 15 minutes have been me hacking around, not egregiously but more randomly than I’d prefer, telling me that I need a break before I start doing actual harm.

Remind me to separate the test file from the init file, since the init might be useful and the tests wouldn’t like to change every time the init gets changed.

I think we have a decent frame in place for FileProvider and INCLUDE, but that they probably both need to be beefed up to deal in some sensible way with errors. If seems likely that raising a few carefully chosen exceptions will bring us back to the Forth prompt in main and return error messages to our headless tests.

I have a general feeling that there is work to be done around running headless, running in tests, and running in main. I could be wrong: main_loop returns with an error message when things go wrong, so it is possible that headless running consists of just running main_loop on one’s init code. But I still don’t like the way we print ‘ok’, and you can see that error messages are coming out twice in the main. That’s odd.

But look at that INCLUDE:

    def define_include(self):
        def _include(f):
            file_name = f.next_token().lower()
            fp = FileProvider(file_name)
            f.main_loop(fp)

        self.pw('INCLUDE', _include)

That’s pretty nice, a Forth word that runs main_loop to do a big job. I have to admire a program that can do that. Anyway, time for a break.

#StopTheCoup! See you next time!