The Robot World Repo on GitHub
The Forth Repo on GitHub

I’m inching toward a World server that I can drive from Python, Forth, or any language. Why is this taking so long? Also: Love one another.

I think we are close to a point where I can almost nearly begin to build a Robot World server that can receive JSON connections via sockets, process them, and display the master PyGame view of the world. But something seems to keep slowing me down, and to defer making progress. I think that I know what the obstacle is. Maybe talking about it will help.

It’s fear.

I freely grant that I don’t know how to do this, although I have done similar things in other languages at other times, most of those times quite some years ago. But what is there to be afraid of? I don’t have to do this, though I have kind of promised that I would. I can try it as many times as I want, and draw lessons to write about every time. I can try it in private, writing about nothing, or something else on that day, if for some reason I don’t want you to see me. And my whole thing here is to do what I do, watch what I do, and draw lessons from it, so what’s the big deal?

I think the big deal is this: I’m well outside my comfort zone here. Faced with almost any single-threaded problem, I can usually imagine a close-enough solution to get started, and using judgment based on experience, and long years or practice, I can refactor from an early solution to a better one. So even if I have never written Sudoku or Space Invaders before, I know pretty much how to do such a thing, so I am happy to evolve from a simple solution to at least a pretty good one. I am really never faced with “I don’t know how to do this”. I might look up some language detail. I might rarely look up an algorithm, like a path-finding routine for a dungeon. But it’s never “I don’t know how to do this”.

I don’t know how to do this. What should I do?

It’s dangerous to go alone.

I am told that in The Legend of Zelda, an old man will say that and give you a sword to take along. What I’d really like to take along with me on this journey is someone who knows the way, someone whose knowledge about sockets exceeds my own, someone with at least a bit more experience than I have. Failing that, someone who I can travel with and who will have ideas and who will have my back.

I do not have that. None of my pals has stepped up to pair with me on this, at least in part because no one I know gets up at the obscene hours that I do, and at least in part because they have their own work to do. And, some of them, because they hate Python. I hope none of them hate me.

I don’t have a sword to take with me, although I do have a large collection of interesting knives. But swords and knives aren’t really the tools I need in programming. And I do have some examples, from the RealPython sockets tutorial. In earlier articles, I’ve looked at their simpler examples and learned a bit. Today, in an attempt to get off the dime, we’ll look at their final example, “Application Client and Server”. I think it may get me close to ready to begin to start.

The Application Server

In this example, RealPython has divided the client code into two pieces, and the server code as well. We’ll start on the server side.

What I’ve gleaned so far is that the basic event-handling code is in the file ‘app-server.py’ (and ‘app-client.py’) and then there is code for dealing with formatting and decoding the byte stream that the ‘app’ part does, called ‘lib-server.py’ [and ‘lib-client.py’]. Let’s talk further with the code in front of us:

We’ll skip the setup, on the grounds that is is essentially boiler-plate, and look at the app-server’s main loop:

sel = selectors.DefaultSelector()

try:
    while True:
        events = sel.select(timeout=None)
        for key, mask in events:
            if key.data is None:
                accept_wrapper(key.fileobj)
            else:
                message = key.data
                try:
                    message.process_events(mask)
                except Exception:
                    print(
                        f"Main: Error: Exception for {message.addr}:\n"
                        f"{traceback.format_exc()}"
                    )
                    message.close()
except KeyboardInterrupt:
    print("Caught keyboard interrupt, exiting")
finally:
    sel.close()

def accept_wrapper(sock):
    conn, addr = sock.accept()  # Should be ready to read
    print(f"Accepted connection from {addr}")
    conn.setblocking(False)
    message = libserver.Message(sel, conn, addr)
    sel.register(conn, selectors.EVENT_READ, data=message)

OK, what do we get here? When key.data is None, we have an incoming connection and we set up a Message instance for it, and we register that message. Message is a class in the ‘lib-server’ file, as we’ll see in a moment. At a guess, but I think a pretty good one, when we get a read event with that connection/address, it will show up as key.data, and we will send process_events to it. Let’s have a glance at that now. It’s a large class, over 200 lines. No wonder I feel a bit trepidatious.

class Message:
    ...
    def process_events(self, mask):
        if mask & selectors.EVENT_READ:
            self.read()
        if mask & selectors.EVENT_WRITE:
            self.write()

    def read(self):
        self._read()

        if self._jsonheader_len is None:
            self.process_protoheader()

        if self._jsonheader_len is not None:
            if self.jsonheader is None:
                self.process_jsonheader()

        if self.jsonheader:
            if self.request is None:
                self.process_request()

    def _read(self):
        try:
            # Should be ready to read
            data = self.sock.recv(4096)
        except BlockingIOError:
            # Resource temporarily unavailable (errno EWOULDBLOCK)
            pass
        else:
            if data:
                self._recv_buffer += data
            else:
                raise RuntimeError("Peer closed.")

That’s plenty for a first look. If our server has data to read, we _read whatever is there. I remember from my reading that we cannot be sure that we get the whole message in one go, so that we may have to read several times to get what we need. And we can get from the shape of the read method that first we get the json header length, then the json header (whatever that may be) and then finally the request.

Looking over at the client side, I begin to see why I feel stymied. Here is the code that creates the json_header:

        jsonheader = {
            "byteorder": sys.byteorder,
            "content-type": content_type,
            "content-encoding": content_encoding,
            "content-length": len(content_bytes),
        }

I can’t keep this many balls in the air all at once. Apparently we need to tell someone—the json decoder?—all this byte order content type and encoding stuff. I get the length part. Let’s go back to the server side and see how it uses that header.

    def process_jsonheader(self):
        hdrlen = self._jsonheader_len
        if len(self._recv_buffer) >= hdrlen:
            self.jsonheader = self._json_decode(
                self._recv_buffer[:hdrlen], "utf-8"
            )
            self._recv_buffer = self._recv_buffer[hdrlen:]
            for reqhdr in (
                "byteorder",
                "content-length",
                "content-type",
                "content-encoding",
            ):
                if reqhdr not in self.jsonheader:
                    raise ValueError(f"Missing required header '{reqhdr}'.")

OK, so far we just make sure we have enough data and then make sure that we have all the headers. No clue how they are used. Maybe that’s in process_request:

    def process_request(self):
        content_len = self.jsonheader["content-length"]
        if not len(self._recv_buffer) >= content_len:
            return
        data = self._recv_buffer[:content_len]
        self._recv_buffer = self._recv_buffer[content_len:]
        if self.jsonheader["content-type"] == "text/json":
            encoding = self.jsonheader["content-encoding"]
            self.request = self._json_decode(data, encoding)
            print(f"Received request {self.request!r} from {self.addr}")
        else:
            # Binary or unknown content-type
            self.request = data
            print(
                f"Received {self.jsonheader['content-type']} "
                f"request from {self.addr}"
            )
        # Set selector to listen for write events, we're done reading.
        self._set_selector_events_mask("w")

Again we wait until we have enough data, using the “content_length” from the header. So that’s what that field is for. Then we check “content-type” to see if it is “text/json”. So we only need that if there is an alternative. We’re going to require json, I reckon, so we can skip that.

And we use the “content_encoding” field as input to our _json_decode:

    def _json_decode(self, json_bytes, encoding):
        tiow = io.TextIOWrapper(
            io.BytesIO(json_bytes), encoding=encoding, newline=""
        )
        obj = json.load(tiow)
        tiow.close()
        return obj

To get the complete drift here, we’ll need to look at io.TextIOWrapper and io.BytesIO, but mostly I want to know what encoding is going to be.

A somewhat confusing search tells me that it comes from the request, which is set up when the Message instance (on the client side) is created. In ‘app-client’ I find this:

def create_request(action, value):
    if action == "search":
        return dict(
            type="text/json",
            encoding="utf-8",
            content=dict(action=action, value=value),
        )
    else:
        return dict(
            type="binary/custom-client-binary-type",
            encoding="binary",
            content=bytes(action + value, encoding="utf-8"),
        )

This example, I am coming to realize, is set up to allow for binary communication and JSON communication. The JSON side will always be ‘utf-8’. We can, I think, simplify a lot of this in our situation, although I imagine one might want one’s server to be somehow impervious to random binary messages.

I’m thinking that if we know that our message is JSON, we can send just a two-bye length, followed by the JSON, not this three layer cake that is in the tutorial.

Reflection

I’m close to nearly having a glimmer of an idea for a spike. It might go like this:

Both server side and client side of our Robot World will need to deal with reads and writes, but only JSON, no binary. So we can start with roughly the event loop of the ‘app’ example we’ve been looking at here. We can follow the lead of the example and provide a Message class that will handle the JSON aspect of reading and writing, including a length but probably not two nested lengths that RealPython used. And we will not call both Message classes Message, to facilitate working with the files from both sides in the same PyCharm project.

Basically we’ll start with something like the boiler-plate from the ‘app’ part of what we’ve just looked at.

We’ll start, I think, from the client side initiating a connection. I’m not sure whether we’ll have it send immediately or wait for a message from the server. In general, our game clients will probably be polled to get their input, but we’ll stay loose on that for a while.

The client will have a canned, hand-crafted message to send to the server, with length and some canned JSON. We’ll have a similar canned reply to send back.

I think we will go with a long-duration basic connection, not reconnecting on each message, just messaging back and forth. We’ll have to deal with dead connections at some point, of course.

So … maybe the client side sends and receives one cycle, but stays alive, sending again on a keystroke or something like that.

Once that works, we can work on creating, encoding, and decoding JSON, until we have enough to run the game. The spike will evolve, or be thrown away, as we may later decide. Either way, what we learn will move forward until we have the game running on sockets.

I can’t even estimate how many sessions this will take. Five? Ten? We’ll know more after we’ve done a bit.

Summary

We have a tentative plan. We’ll start actually executing the plan in the next article. I’m not sure if that will be today or tomorrow.

Please join me then. And love one another.