Next Evolution
The Robot World Repo on GitHub
The Forth Repo on GitHub
I’m glad I decided to take a break, because I have some smaller steps in mind today that I didn’t have yesterday. Also: Love one another.
Personally, I find that after a couple-three hours of coding and writing, my brain is usually tired, and usually locked in one whatever I’ve been working on. The bigger picture isn’t always so clear, as if the detailed work has flushed those buffers. And, while sometimes I see what the next small steps might be, quite often I do not. Then, after a break, often overnight, I have a much better view the next time around. This is one of those times.
Our current server code is very rudimentary, basically copying input bytes into the output buffer and writing them back to the client. This might be why it is called “Echo Server” in the RealPython tutorial I borrowed it from.
What we actually need is for the server to receive a message, not just raw bytes, and to submit that message to our application (presumably the World side of Robot World), and to have the application produce a reply message. So the first question before us is “how do we know when we have the whole message”. And the answer is: “we will prefix the message with its length”.
Among the steps I think we’ll take this morning, we may find:
-
Prefix our test client messages with their lengths. This should not bother the server at all, since it is just echoing. We’ll have to adjust the tests’ expectations, of course.
-
In the server code, maintain enough state so that we know whether to read the length, and do so.
-
In the server code, after the length has been read, ensure that we have at least that many characters available in the input, and only then echo it.
-
Enhance what we echo, first perhaps with a prefix like “I saw”, moving toward a small set of predefined input messages and canned replies. Rinse, repeat, refine ideas …
As things stand, the server uses a SimpleNamespace to hold its buffers. I am curious about that: couldn’t it just as well have been a dictionary? Probably. We’ll try to avoid that path … but what we will do, I think, is to move to some other object, perhaps MessageHandler, to do our counting and replying.
I am aware that there is a huge 200 line solution to this on RealPython. We’re going to try to roll our own, although we might look at RealPython for details. Let’s get started with some length work. Cribbing from the tutorial, I find that we use struct.pack and struct.unpack for that purpose. Let’s write a little test:
def test_pack_unpack(self):
value = 1234
packed = struct.pack('>H', value)
seq = struct.unpack('>H', packed)
assert seq[0] == value
I’m glad that I wrote that, because unpack returns a sequence, and I had not noticed that.
Now let’s have a little function to prefix a message:
def test_one_go_round(self):
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
s.connect(("127.0.0.1", 34567))
msg = b"Hello, world"
prefixed = prefix(msg)
s.sendall(prefixed)
data = s.recv(1024)
def prefix(msg):
num = len(msg)
prefix = struct.pack('>H', num)
return prefix + msg
We are green. Commit: testing prefix creation.
Now I think we’ll create a trivial MessageHandler object to use instead of the SimpleNamespace used here in our server:
class WorldServer:
def accept_wrapper(self, sock):
conn, addr = sock.accept() # Should be ready to read
print(f"Accepted connection from {addr}")
conn.setblocking(False)
data = types.SimpleNamespace(addr=addr, outb=b"")
events = selectors.EVENT_READ | selectors.EVENT_WRITE
self.sel.register(conn, events, data=data)
I plan to just type it in, let the tests break, then have them work again in their present form. That turns out to be trivial:
class MessageHandler:
def __init__(self, addr, inb, outb):
self.addr = addr
self.inb = inb
self.outb = outb
def accept_wrapper(self, sock):
conn, addr = sock.accept() # Should be ready to read
print(f"Accepted connection from {addr}")
conn.setblocking(False)
data = MessageHandler(addr=addr, inb=b"", outb=b"")
events = selectors.EVENT_READ | selectors.EVENT_WRITE
self.sel.register(conn, events, data=data)
No surprise but isn’t that a delightfully small step? Commit: new MessageHandler object.
Now, we’d really prefer that the WorldServer class would keep its hands out of MessageHandler’s pockets. Let’s pass it the input, and request the output. That means that we’ll want to change this:
def service_connection(self, key, mask):
sock = key.fileobj
data = key.data
if mask & selectors.EVENT_READ:
recv_data = sock.recv(1024) # Should be ready to read
if recv_data:
print(f'Received {recv_data}')
data.outb += recv_data
else:
print(f"Closing connection to {data.addr}")
self.sel.unregister(sock)
sock.close()
if mask & selectors.EVENT_WRITE:
if data.outb:
print(f"Echoing {data.outb!r} to {data.addr}")
sent = sock.send(data.outb) # Should be ready to write
data.outb = data.outb[sent:]
I think we’ll try sending the data into MessageHandler on a read call, like this:
def service_connection(self, key, mask):
sock = key.fileobj
data = key.data
if mask & selectors.EVENT_READ:
recv_data = sock.recv(1024) # Should be ready to read
if recv_data:
print(f'Received {recv_data}')
data.read(recv_data)
Ah, I don’t like this … I think we need to do the reading and writing in the handler. Back that out and try a different path.
I think our MessageHandler will need to be created with more information than we presently give it, which is really none. Let’s make that explicit, just passing the address for some reason that I do not understand.
class MessageHandler:
def __init__(self, addr):
self.addr = addr
self.inb = b""
self.outb = b""
Green, commit: don’t pass buffers in on creation.
I’ll try to move all the reading into MessageHandler. We’ll have to at least pass in the socket. Basically I plan to move the READ code into MessageHandler and make it work there. Let’s extract a method locally, like this:
Wait, I want to remove that print that accesses addr.
def service_connection(self, key, mask):
sock = key.fileobj
data = key.data
if mask & selectors.EVENT_READ:
self.process_read(sock, data)
if mask & selectors.EVENT_WRITE:
if data.outb:
print(f"Echoing {data.outb!r} to {data.addr}")
sent = sock.send(data.outb) # Should be ready to write
data.outb = data.outb[sent:]
def process_read(self, sock, data):
recv_data = sock.recv(1024) # Should be ready to read
if recv_data:
print(f'Received {recv_data}')
data.outb += recv_data
else:
# print(f"Closing connection to {data.addr}")
self.sel.unregister(sock)
sock.close()
Now let’s put that method into our MessageHandler, almost as is. It doesn’t quite want to work that way. It will work like this, I think:
class WorldServer:
def service_connection(self, key, mask):
sock = key.fileobj
data = key.data
if mask & selectors.EVENT_READ:
data.process_read(self.sel, sock)
if mask & selectors.EVENT_WRITE:
if data.outb:
print(f"Echoing {data.outb!r} to {data.addr}")
sent = sock.send(data.outb) # Should be ready to write
data.outb = data.outb[sent:]
class MessageHandler:
class MessageHandler:
def __init__(self, addr):
self.addr = addr
self.inb = b""
self.outb = b""
def process_read(self, sel, sock):
recv_data = sock.recv(1024) # Should be ready to read
if recv_data:
print(f'Received {recv_data}')
self.outb += recv_data
else:
# print(f"Closing connection to {data.addr}")
sel.unregister(sock)
sock.close()
We are green. Commit: move read handling to MessageHandler.
I don’t like WorldServer referring to MessageHandler as data. Let’s improve that.
def service_connection(self, key, mask):
sock = key.fileobj
handler = key.data
if mask & selectors.EVENT_READ:
handler.process_read(self.sel, sock)
if mask & selectors.EVENT_WRITE:
if handler.outb:
print(f"Echoing {handler.outb!r} to {handler.addr}")
sent = sock.send(handler.outb) # Should be ready to write
handler.outb = handler.outb[sent:]
Commit: rename temp variables for clarity.
Now it seems to me that we can do the same kind of thing, moving the write to the handler. Then we’ll reflect on why we did this.
class WorldServer:
def service_connection(self, key, mask):
sock = key.fileobj
handler = key.data
if mask & selectors.EVENT_READ:
handler.process_read(self.sel, sock)
if mask & selectors.EVENT_WRITE:
handler.process_write(sock)
class MessageHandler:
class MessageHandler:
def __init__(self, addr):
self.addr = addr
self.inb = b""
self.outb = b""
def process_read(self, sel, sock):
recv_data = sock.recv(1024) # Should be ready to read
if recv_data:
print(f'Received {recv_data}')
self.outb += recv_data
else:
# print(f"Closing connection to {data.addr}")
sel.unregister(sock)
sock.close()
def process_write(self, sock):
if self.outb:
print(f"Echoing {self.outb!r} to {self.addr}")
sent = sock.send(self.outb) # Should be ready to write
self.outb = self.outb[sent:]
Green. Commit: reading and writing done in MessageHandler.
Reflection
I think we might take our break here. Let’s reflect.
In a few small steps and maybe one or two missteps, we have moved the reading and writing out of WorldServer, into MessageHandler. Why have we done that?
It seems to me to be a step toward what we need, which is to have our responses not just echo the inputs. We are now in a position, for example, to read a length field and ensure that we have the whole message. Then, we can “explicitly” format our response, which will at first still just be an echo. But in MessageHandler, we have a place to stand where we can compute a different answer. At first perhaps by hand, then perhaps with a lookup in a dictionary of calls and responses, … and finally calls to World to process actual commands.
What we’ll do, roughly, is buffer our received bytes into inb, and when we have enough, copy or otherwise produce the output in outb. The write operation will be called all the time, but with no bytes to write, we’ll just return. Perhaps later we’ll fiddle with the selector so that we don’t get any write calls until we want to see one.
We will surely have to modify our tests along the way, providing new ones, possibly deleting old ones. We are changing the protocol of our object (the messages it receives) so we must expect that additional bit of hassle.
But we are on our way, with a separate object serving as the place to begin to make output differ from input. I think we’ve made a decent step this morning, and we’ll enjoy a nice iced chai in celebration.
See you next time. Love one another, even those who are not quite like you.