Toward Client-Server?

The Repo on GitHub

This morning, without any direct Python network communications experience, I plan to start pushing the bot world code toward client-server. Along the way, I’m going to think about what GeePaw Hill means by a Making App. We do make a small but nice bit of progress.

As things stand now, the only “main program” we have is the pygame one that spawns a bunch of Bots into a World and lets them drive around collecting Blocks. The only view we have is in that pygame main, and it displays the whole world’s status, and very naively at that. In actual use as a game or competition or whatever it is, each client would be a program that fields a number of bots (and whatever else we ultimately allow them to field). The client would connect to a single world server which could provide that client either with a private world or a community world, depending on where we go. The client program would—I assume—or tentatively design—produce a single “message” of commands for each of its bots, and would send that to the world server, where it would be processed. The world server would then return a “message” containing whatever update information we deem appropriate. Today that would be things like the position of each bot, its vision of its immediate surroundings, and the scent of the area.

Note: I think the current information is too much focused on our current particular ridiculously simple bots. But there will be some kind of information message coming back, in a defined format that the client code can interpret as it wishes.

In the current code, the Bot has a member variable world which is in fact an instance of World class, the one that is running the simulation. So our game main creates a World, and a bunch of Bot and Blocks, stuffs them into the world. The game then repeatedly cycles through all the bots, sending them do_something in turn, and all the bots talk to the world and get updated.

I think—tentative design again here—that each bot cannot be allowed to send and receive network messages to the server on its own hook, because even an optimistic estimate of the turnaround time for a message would mean that with a large number of bots making a couple of decisions per turn, things would get too slow and too network-heavy. So my overall plan is that there will be a top-level client object that holds all the bots and cycles them, and that the bots will return their desired actions as little request data packets. The top-level accumulates all these requests and sends them all to the server, receiving back a packet of responses, which it distributes to the bots, rinse repeat.

So that’s what I’m going to work on today. But first, I want to think about GeePaw’s notion of a “Making App”, mostly to express it to myself, and I hope you’ll enjoy the ride.

GeePaw describes developing a product using two “apps”, a “Making App” and a “Shipping App”. His quick definition, you read it here first, hot off the presses, is this:

In a client/server world, be the client http, desktop, or mobile, I see developers spend one helluva lotta time using the client for their own wicked non-customer purposes. Clients are tuned for customer usage, not developer usage. Some symptoms of that:

Devs firing up client and firing up server to work, then changing one, then forgetting to fire it up again, then changing the other, then having no effect, then wondering if they fired it up again.

Rapid-fire page-banging to get to the part of the client they want to see.

Inability to do simple dev tasks like reset to the starting data, or change what time it is in the universe.

Binding of whole development teams to the three instances of the multi-service rig.

Burning clock cycles waiting for async transport mechanisms.

Flipping through security gates that are not relevant to their current goal.

A making app is an app we make that uses the server code and uses the client code but

a) adds some other code and
b) repackages it into a single desktop app that is tuned for developers, not users.

I do not fully understand these ideas, but what GeePaw has said is interesting, he thinks we need a Making App here in our robot game, and I want to try it. Here’s what I would say if someone said “tell me what you know, surmise, guess, and fantasize about GeePaw’s notions of Making App and Shipping App”:

Shipping App: We’re building some application. The Shipping App is a build of the application suitable for deployment. It might have multiple parts, a server part that gets put on the Internet somewhere, and client parts that are sent to users. these might be Android or iOS apps, or executable programs to run on their computers, or some kind of web interface. The Shipping App quite likely has a database component of some kind, might spin up micro-services, all the things that today’s applications do. It probably operates over the Internet in some kind of client-server or peer-to-peer networking.
Making App: The Making App is a build intended for development: for making the application. It is configured to work on a developer’s machine, not requiring a server, a connection to Oracle (ptui!), the ability to spin up AWS servers, and so on. And yet, it supports all the things that the application does, using—I’m guessing even more here—proxies for the network, the database, and so on. Everything trimmed down, running on my machine. The Making App includes tools, viewers, logging, inputs and outputs that let the developer work on the app, test it efficiently, find out what it is doing, and progress the work as rapidly as possible.
Example: I’m pretty confident in this example, based on conversations with GeePaw. Suppose the app is a client-server application. So, there will be two very separate “main” programs, the client and the server. Down at the bottom of those programs, there will be some network communication code, sockets or whatever scheme is used for the two to communicate. A bit above that layer, there is a “language” of textual messages that the two components send back and forth, the request format and the response format. These belong to the application, and what’s below them is infrastructure and should ideally have nothing to do with the specific application. Separation of concerns.; The Shipping App will have all the real sockets code in it. The Making App will (probably) not. Instead it will have some code stubs, matching the interface to the sockets code, but the stubs will just pass the request string over to the server and the server will just pass the response string back. It happens instantaneously, no network interaction, and all in one executable.

OK, the above is very speculative. I’ve never knowingly written a Making App, though I have written innumerable programs and tools for developing them. I’ve probably done most of the things one does in a Making App. But I do not know what GeePaw means by the idea, and I do not know what ideas he has that let him call out this notion so clearly. I hope to find out as we go forward.

But meanwhile, I have to get some work done, and with or without a Making App, I want to make some progress toward client-server. Here is my Cunning Plan, p-baked for p around 0.25

p(Cunning Plan) = 0.25 ± ε

There are requests that the Bot can make for things to happen in the world. I’ve been calling them commands but “requests” is probably a better word. The bot can request a “take” or a “drop” or a “step”. the valid requests are up to the designer of the world, but they are part of an agreed “language” between the world server and the clients.

There are requests that the client can make, that will let it introduce a bot to the world, get it placed somewhere, so that henceforth it can send that bot’s requests to the world and get the responses back.

Requests and responses will be textual, in some hopefully convenient form, CSV or JSON perhaps¹.

Therefore, I envision a new object on the Bot side of things, that looks like the World to that side of things, but that is part of the client. It will contain all the bots, cycle them with do_something or its later equivalent, accumulate their requests, send them to the world, receive the response, and distribute the answers. Some of those answers will be sent to the bots, and some will belong to this new object.

I envision what we have now as a direct link between World and Bot:

World <---> Bot

That’s because that’s what it is. It’s true that we are now returning a string request from the state machine, such as take or drop but that’s actually still managed in the Bot:

class Bot:
    def do_something(self):
        self.update_knowledge()
        self.state = self.state.update(self._knowledge)
        self.do_state_actions()
        self.move()

    def do_state_actions(self):
        for action in self.state.action(self._knowledge):
            match action:
                case 'take':
                    self.world.take_forward(self)
                case 'drop':
                    self.world.drop_forward(self, self.inventory[0])
                case _:
                    assert 0, f'no case {action}'

So there is a moment where we have a string request but we instantly resolve it into a direct message to world. And all the information that comes back, presently, is sent directly from World back to the individual Bot:

class World:
    def take_forward(self, bot: Bot):
        take_location = self.bots_next_location(bot)
        if take_location == bot.location:
            return
        entity = self.map.entity_at(take_location.x, take_location.y)
        if self.can_take_entity(entity):
            self.map.remove(entity.id)
            bot.receive(entity)

That line bot.receive(entity) is a direct call from World to Bot. That won’t do in the networked scheme: we have no memory access to the bot, we cannot send it a direct message.

Now I can see at least two ways we might go:

We might replace bot.receive(entity) with something that creates a message and appends it to the response message somehow;
We might replace the bot instance with some kind of BotProxy that understands messages like receive and creates the response message components and accumulates them, until they are collected by an overall process that is cycling through the full client message and building up the response.

I have seen a lot of code that worked like #1 and I didn’t like it. It had details of the messages and formats all over the program. So I am leaning toward a structure that is more like this:

World <--> BotProxy <--> ClientProxy <--> Network <--> WorldProxy <--> Bot

Something like that, it seems to me, would let the various objects think in terms of the other objects they are really interested in. The World is interested in Bots and what they ask for, and it wants to give them back the results. It doesn’t want to think about converting what it knows into strings. Similarly, the Bot really doesn’t want to return a string ‘drop’ to issue a drop request. It was happier saying to the world: world.drop.

This is, I’m afraid, going to get labelled as “speculative” and someone is going to say YAGNI at me and ask me why I can’t evolve from where we are to that. To them I say that I was there when YAGNI was invented and may have even invented the acronym myself, and this is prediction, not speculation. However, I do want to evolve toward the eventual solution, not build up a raft of complicated objects that may not be what we need.

I think the first thing we need is an object on the client side that holds all the Bots and cycles through them. We’ll pass that to the World and have the World understand what it has to to keep things working.

I think this new object is the Population. Let’s TDD a little something up, almost a spike in character just to see what a Population might feel like.

class Population:
    pass


class TestPopulation:
    def test_exists(self):
        population = Population()
        assert population is not None

So far so good. I imagine that we’ll have some way of specifying what the population should be but for now I think just a number of bots would be good.

class Population:
    def __init__(self, bot_count=0):
        self._bots = []
        for i in range(bot_count):
            self._bots.append(Bot(i, i))

    @property
    def bots(self):
        return self._bots


class TestPopulation:
    def test_exists(self):
        population = Population()
        assert population is not None

    def test_has_bots(self):
        population = Population(2)
        count = 0
        for bot in population.bots:
            count += 1
        assert count == 2

So that’s nice. I think that the Population will ultimately have the link to the WorldProxy, so for now the world. How do we get one now? Ah, the tests actually create one and populate it. So our new Population will just do that, for now. How can we test that? I think we need some capability to give Population some testing into and get it back. In fact, I’m starting to think that Population should be a fairly dumb collection of Bots and no more than that and that we need a new object to deal with it. So be it. Let’s make Population into a collection. I’ll do this in the dullest way that I know.

class Population:
    def __init__(self, bot_count=0):
        self._bots = []
        for i in range(bot_count):
            self._bots.append(Bot(i, i))

    def __iter__(self):
        return iter(self._bots)


class TestPopulation:
    def test_exists(self):
        population = Population()
        assert population is not None

    def test_has_bots(self):
        population = Population(2)
        count = 0
        for bot in population:
            count += 1
        assert count == 2

OK, that’s it for Population, almost.

Now there is a very odd thing in the program now, which is this:

class World:
    def add(self, entity):
        entity.world = self
        World.next_id += 1
        entity.id = World.next_id
        self.map.place(entity)

Note that World jams two values into the entity (Bot or Block) passed to it, their id and a link back to the world.

I really want to evolve to this. But we’ve gone so far in linking these two sides together that it is difficult to see a way out. What can we do in a step-by-step fashion? Our tests do things like this:

    def test_step(self):
        world = World(10, 10)
        bot = Bot(5, 5)
        world.add(bot)
        location = bot.location
        bot.direction = Direction.NORTH
        bot.step()
        assert bot.location == location + Direction.NORTH

OK, small steps. And just a few more steps left today, time runneth out.

I want a WorldProxy that contains the Population and runs it. I want the WorldProxy to know the world. For now, we’ll give it the world. We want our Entities to know the proxy, not the real world. We’ll solve that problem in a horrible way at first: we’ll re-jam the value that World jams into our Bots.

I’m not ready to commit to these objects yet, so I’m going to keep testing in this one file, test_population.

class TestWorldProxy:
    def test_exists(self):
        proxy = WorldProxy()

    def test_add_bot(self):
        proxy = WorldProxy()
        bot = Bot(5, 5)
        proxy.add(bot)
        assert bot.world == proxy

Hm, that’s not too far off from what we might want. Let’s make it work.

class WorldProxy:
    def __init__(self, width=20, height=20):
        self._population = Population()
        self._world = World(width, height)

    def add(self, entity):
        self._world.add(entity)
        entity.world = self
        self._population.add(entity)

class Population:
    def __init__(self, bot_count=0):
        self._bots = []
        for i in range(bot_count):
            self._bots.append(Bot(i, i))

    def __iter__(self):
        return iter(self._bots)

    def add(self, entity):
        self._bots.append(entity)

Tests are green. We have added the bot to a world, stuffed it in our Population, and so far so good. I’m going to commit this, it’s just a test file. Commit: initial Population and WorldProxy.

Let’s see if we can do a round trip. I think we can.

    def test_round_trip(self):
        proxy = WorldProxy()
        bot = Bot(5, 5)
        proxy.add(bot)
        proxy.run_cycle()
        assert bot.location == Location(6, 5)

I just made up the name run_cycle because it’s barely better than do_something.

class WorldProxy:
    def run_cycle(self):
        for bot in self._population:
            bot.do_something()

This probably fails sending messages to WorldProxy that World would understand. Let’s find out.

    def step(self):
>       self.world.step(self)
E       AttributeError: 'WorldProxy' object has no attribute 'step'

Exactly! So let’s implement that to forward to world.

class WorldProxy:
    def step(self, bot):
        self._world.step(bot)

Still fails and I wonder why.

Expected :Location(6, 5)
Actual   :Location(5, 4)

Ah that’s interesting, I thought we were facing East. It appears not. That looks like we moved north.

Let’s set the direction:

    def test_round_trip(self):
        proxy = WorldProxy()
        bot = Bot(5, 5)
        bot.direction = Direction.EAST
        proxy.add(bot)
        proxy.run_cycle()
        assert bot.location == Location(6, 5)

Test passes. I do wonder why.

class Bot:
    def __init__(self, x, y, direction=Direction.EAST):
        self.world = None
        self.id = None
        self.name = 'R'
        self.direction_change_chance = 0.2
        self.tired = 10
        self._knowledge = Knowledge(Location(x, y), direction)
        self.state = Walking()

That sure looks to me like it would have set the direction to EAST. Let’s assert.

I put in an assert and it passes. I remove the assert and go back to the original test and it passes. It’s as if we got NORTH just once and never again, and I feel sure that I didn’t change anything. I run the test repeatedly and it fails sometimes. This does not please me.

I put my assert back. Test looks like this:

    def test_round_trip(self):
        proxy = WorldProxy()
        bot = Bot(5, 5)
        assert bot,direction == Direction.EAST
        proxy.add(bot)
        proxy.run_cycle()
        assert bot.location == Location(6, 5)

PyCharm has squiggles under direction, and the popup says Unresolved reference direction. WTF, over? And the assertion always passes and the test sometimes fails. Why does the test file think direction is unresolved and is that related to this issue at all?

On the off chance, I restart PyCharm. I debug. Oh! a Bot has a built-in 20 percent chance of changing direction! Doh! We have to override that.

    def test_round_trip(self):
        proxy = WorldProxy()
        bot = Bot(5, 5)
        bot.direction_change_chance = 0
        proxy.add(bot)
        proxy.run_cycle()
        assert bot.location == Location(6, 5)

Solid green. And it’s is 0957 and I started at 0657, so it is well past time to stop. Commit: test ran a bot round trip no take or drop just step.

Summary

We’ve taken the first few small stumbling steps in the direction of client-server. We have a Population collection of bots, which should probably be entities or perhaps two populations, and we have a WorldProxy object that uses Population to hold a bot and run a round trip on them.

Possibly we should push the do_something down into the Population. It’s not carrying much weight right now, but experience suggests that once we build a specialized collection it will turn out to be useful. The WorldProxy intercepts the Bot’s call to world to step but is not yet intercepting the reply. We’ll work on that the next time we’re fresh. I envision at least two ways to go, similar to what we spoke of above, either with a proxy for the World to speak with, or by having the World return some values to be stuffed into the Bot’s knowledge.

I like the proxy idea, because I like the idea of just talking to the other object. But it may become too confusing with proxies all over. Once it’s all unwound, however, I think it will be just like talking to the other objects, which is pretty much what I like. Note to self, however: we can send them messages but we cannot expect returns. Those must come around later with messages back to us. It seems very recursive, but it isn’t.

Emphasis: I want to emphasize that these two tiny classes, which may seem like almost nothing, do begin to express my understanding of how the client-server code will work. There will be some kind of interface+collection on the client side, that accumulates messages and sends them to the server, and the server will process those messages, each of which will typically represent request involving specific individual entities in the server’s map.; We now have a place to stand on the client side, with Population and WorldProxy, where the accumulation, communication, and dispersal of information can take place. It’s very small, and it’s surely not entirely right, but it is a tiny step in what seems to me to be the right direction.; We can only ever work in small steps, typing one statement at a time. The trick is how frequently our tests get to green, with the code in a form we can commit. We can work one statement at a time for a month, or for a few minutes. Committing once a month is troublesome. Committing once every few minutes really isn’t. It’s true that we didn’t implement client server, or even a Making App today. We just took steps in that direction. That’s what we do.

See you next time!

JSON is not really “convenient” by my lights, but it is better than XML, I suppose. I guess it’s fine if you don’t ever have to read it or write it by hand. I am told that Python’s JSON is easy to use but I have not tried it. ↩