Before taking our new message format forward, let’s think about how we almost missed the improvement. “It’s dangerous to go alone. Take this.”

Our message format from client to server, so far, is a simple list of tiny dictionaries with keys ‘entity’, ‘verb’, and ‘param1’. Until yesterday it was much more complicated, consisting of a list of requests, each of which was a dictionary with keys ‘entity’ and ‘actions’, with ‘actions’ pointing to a list of tiny dictionaries with keys ‘verb’ and ‘param1’.

I got to that larger structure by thinking that in action, we’d process one entity, with multiple actions, because that’s literally what happens: a given Bot can turn, step, take or drop, all in one turn. We don’t even prohibit multiple steps if a Bot were to try that. So it seemed natural to think of an “EntityRequest” object, holding one entity identifier and a list of actions. And, since the client can certainly have a number of bots, it “made sense” that we’d have a list of Bots, and therefore get a list of these Requests.

And then, it “made sense” that we’d send that list of dictionaries of lists of dictionaries over to the World to be processed. And thereby hang a half-dozen or more articles.

Many years ago, Kent Beck used to ask us “What is the simplest thing that could possibly work?” And in those days, I took up a slightly different mantra, “Do the simplest thing that could possibly work.” These two ideas are not the same.

Beck’s question invites a conversation with plenty of thinking involved. Often we’d go from “Well, an string might work”, to “no, but what if we had a list of strings”, to “how about an object containing item SKU and quantity”, working from a weak idea to a stronger one.

My mantra is very different in effect. It suggests that we should actually start with the string. Soon, in the code, we’d find that the string wasn’t good enough and we’d refactor to something better.

I can’t really tell you why, in those days, I liked that idea. I can tell you that, since the simplest idea is hardly ever going to work for the long term, plenty of people told me how that idea was inefficient, stupid, and so was I. Probably they were correct.

However, the approach of starting with something simple and evolving it has led to the thousands of articles on this site, which demonstrate, time and again, that we can move to code quickly with whatever ideas we have, never pausing our thinking of course, and that we can refactor to just about any other needed structure, so long as we keep what we have sufficiently neat and clean, removing duplication and creating abstractions as they come to us.

Often in these articles, I start with something too simple. In the current robot world, I started with the Bot calling the world and saying ‘step’ or something about that simple. Clearly did not hold water for the long term, and yet, here we are, with a much more capable and powerful structure, all done in small steps.

Yeah, but what’s your point?

My point is that despite my efforts to keep things simple, I very often make things too complicated. I see a pattern, or some kind of generality, and I go for it. There’s another mantra that I popularized: YAGNI: You Aren’t Gonna Need It.

This one, too, came from Kent Beck: when someone said “we’re gonna need a Frammis, we should build it now”, he would interrupt with “You aren’t going to need it”. His purpose was to draw us back to attention on what we need now, not what we’ll need later. That focus is much like the “simplest thing” idea: we build good code for today and refactor as needed for tomorrow.

Someone: “I have a question about that.”

Ron: “We’ll come back to that.”

But I, Ron, YT, dweller chez Ron, have immersed myself for decades in learning about complex structures in code and data. And, despite plenty of evidence to the contrary, I often think of myself as wicked smart, and I often enjoy building really nifty elegant complex things that do something in an amazing and somehow beautiful way.

The result of that misspent youth and adulthood is that, very often, I create something that is more complicated than the current situation calls for. I do not do the simplest thing, I do not listen to YAGNI. I forget. I get target fascination. I see something really cool and set out to do it.

And I work almost exclusively alone. I do not have a team to help me with my code. I don’t even have a pair. I no longer even have a cat.

If I had had someone working with me, they might have said “Dayum Ron, that’s complicated, can’t we just have a list or something?” If they had, I hope I’d have listened and if I had, we might have found a simple solution sooner.

It’s dangerous to go alone.

Hand offering rubber duck "It's dangerous to go alone".

Maybe I should put a rubber duck on the desk. But, cute as they are, rubber ducks are no substitution for another person or team working with us to get things done simply, yet well.

I’ll try harder to compensate.

Ron: You had a question?

Someone: Yes. Isn’t doing something simple and then refactoring to get what you really needed all along less efficient than just doing what you need?

I’m glad you brought that up. I’d just briefly make two points:

First, it does seem credible that it would be more efficient to do the right thing in one step instead of doing something less than right in one step and then the right thing in another step. But we do not know that, and there are examples that point the other way. I recall one time when my father thought he would save time by not turning off the electricity while he changed out a switch … I’ll leave it to you to think of better examples.

Second, if a partial solution takes less time and works for the moment, we can deliver visible value sooner, and whether we ship that value to users, or just use it to show that we’re progressing, so that our bosses keep the lights on,, early delivery of visible value is a very important strategy.

Third (I was mistaken about the “two”), we cannot really just do the big final solution all in one go. We can only type one line at a time, so we are going to build up whatever we build by application of many small steps. Presumably, since we’re good programmers, we’ll have many objects and methods or functions and mopeds in our final design, and we’ll do them in some sequence. With well-structured code, we can have a partial solution for some part that will ultimately have to be large and generalized and robust, and that partial solution can help us see the shape of the whole program better.

Fourth (I was very mistaken), speculation is not our strongest suit, and more than occasionally our initial ideas about what “we’re gonna need” are flat wrong. We are dealing with that situation right now. I was 90 percent of the way through building the “right” solution to our message format when it finally became clear that the simple solution is quite good enough.

Bottom line, I think it’s an open question which approach delivers the final result sooner, and that it’s clear that getting something simple and clean working early is better than getting nothing working early. Your views may vary, and you’re free to work as you and your employer wish. My employer and I are quite happy this way.

But we digress.

Right, we have our new list of id, verb, parameter working. Yesterday we had this list of things we still need:

  • Make it easier to create the little action dictionaries;
  • Provide and use an entry in DirectConnection to take a list of many actions;
  • Look for duplicated code in DirectConnection and take advantage of it;
  • Cause World to return all the knowledge dictionaries for all the entities in an execute list;
  • Deal with errors, including avoiding exceptions and passing error info in the knowledge.

Let’s have a look at DirectConnection. That feels right-sized this morning.

class DirectConnection:
    def __init__(self, world):
        self.world = world

    def add_bot(self, x, y, direction=Direction.EAST):
        bot_id = self.world.add_bot(x, y, direction)
        result_dict = self.world.fetch(bot_id)
        client_bot = Bot(x, y)
        client_bot._knowledge.update(result_dict)
        return client_bot

    def update_client(self, client_bot):
        result_dict = self.world.fetch(client_bot.id)
        client_bot._knowledge.update(result_dict)

    def step(self, cohort, client_bot_id):
        rq = [{'entity': client_bot_id, 'verb': 'step'}]
        self.world.execute(rq)
        result_dict = self.world.fetch(client_bot_id)
        cohort.update(result_dict)

    def take(self, cohort, client_bot_id):
        rq = [ {'entity': client_bot_id, 'verb': 'take'}]
        self.world.execute(rq)
        result_dict = self.world.fetch(client_bot_id)
        cohort.update(result_dict)

    def drop(self, cohort, client_bot_id, holding_id):
        rq = [ {'entity': client_bot_id, 'verb': 'drop', 'param1': holding_id}]
        self.world.execute(rq)
        result_dict = self.world.fetch(client_bot_id)
        cohort.update(result_dict)

    def turn(self, cohort, client_bot_id, direction_string):
        rq = [ { 'entity': client_bot_id, 'verb': 'turn', 'param1': direction_string}]
        self.world.execute(rq)
        result_dict = self.world.fetch(client_bot_id)
        cohort.update(result_dict)

I believe I see a bit of duplication here. Let’s eliminate that. I predict that we’ll be glad we did: we usually are.

    def step(self, cohort, client_bot_id):
        rq = [{'entity': client_bot_id, 'verb': 'step'}]
        self.run_request(cohort, client_bot_id, rq)

    def take(self, cohort, client_bot_id):
        rq = [ {'entity': client_bot_id, 'verb': 'take'}]
        self.run_request(cohort, client_bot_id, rq)

    def drop(self, cohort, client_bot_id, holding_id):
        rq = [ {'entity': client_bot_id, 'verb': 'drop', 'param1': holding_id}]
        self.run_request(cohort, client_bot_id, rq)

    def turn(self, cohort, client_bot_id, direction_string):
        rq = [ { 'entity': client_bot_id, 'verb': 'turn', 'param1': direction_string}]
        self.run_request(cohort, client_bot_id, rq)

    def run_request(self, cohort, client_bot_id, rq):
        self.world.execute(rq)
        result_dict = self.world.fetch(client_bot_id)
        cohort.update(result_dict)

Better, by my lights. Commit: tidying.

Now those requests are hard to read and create. Let’s make them all alike.

    def step(self, cohort, client_bot_id):
        rq = [ {'entity': client_bot_id, 'verb': 'step', 'param1': None}]
        self.run_request(cohort, client_bot_id, rq)

    def take(self, cohort, client_bot_id):
        rq = [ {'entity': client_bot_id, 'verb': 'take', 'param1': None}]
        self.run_request(cohort, client_bot_id, rq)

Now I wonder if we can extract a method in the middle of the list. PyCharm doesn’t understand what I want. I’ll just code it: I don’t see a clever series of refactorings that will do it.

    def action(self, client_bot_id, verb, param1):
        return {'entity': client_bot_id, 'verb': verb, 'param1': param1}

PyCharm quickly figured out what I wanted and filled in 2/3 of the dictionary there. Very nice.

Use it:

    def step(self, cohort, client_bot_id):
        rq = [ self.action(client_bot_id, 'step', None)]
        self.run_request(cohort, client_bot_id, rq)

Honestly, I don’t like that better. Undo that bit. Sometimes we try things and don’t like them after all. It’s OK to try and it’s really good to undo the things we don’t like.

Well. I think DC is better now, but I had thought it would lead us somewhere better than it did. It’s still too fragmented and it will be until we do some work in Bot or thereabouts. Let’s look at World and its execute.

In World, I find the old process method and the old method it calls. I think when we deleted those tests yesterday we left those unreferenced. Remove, commit: remove old unused process code.

Now, we have decided, if I recall, that when World executes a list of actions, it will automatically return the knowledge dictionaries for all the bots who took action. Let’s do that. Here’s the execute we now use:

    def execute(self, actions_list):
        for action in actions_list:
            id = action['entity']
            entity = self.entity_from_id(id)
            verb = action['verb']
            try:
                param1 = action["param1"]
            except KeyError:
                param1 = None
            self.execute_action(entity, verb, param1)

What we need to do is to keep track of all the entity ids we process, and fetch them all and return that list or set or whatever it is as our result. We can do that with impunity, because no one is looking to this method to return anything right now. None of the existing tests for execute process more than one bot. Let’s quickly gin one up:

    def test_returns_results(self):
        world = World(10, 10)
        bot_1_id = world.add_bot(5, 5)
        bot_2_id = world.add_bot(7, 7, Direction.NORTH)
        rq =  [
            { 'entity': bot_1_id, 'verb': 'turn', 'param1': 'NORTH'},
            { 'entity': bot_1_id, 'verb': 'step'},
            { 'entity': bot_2_id, 'verb': 'step'},
            { 'entity': bot_2_id, 'verb': 'step'},
        ]
        result = world.execute(rq)
        assert len(result) == 2

That’s enough to fail, and it does, because execute doesn’t return anything.

    def execute(self, actions_list):
        ids_used = set()
        for action in actions_list:
            id = action['entity']
            ids_used.add(id)
            entity = self.entity_from_id(id)
            verb = action['verb']
            try:
                param1 = action["param1"]
            except KeyError:
                param1 = None
            self.execute_action(entity, verb, param1)
        return [ self.fetch(bot_id) for bot_id in ids_used ]

Green. Commit: execute now returns knowledge dictionaries for all ids processed.

That test isn’t really very robust, however: we could have returned two integers and it would pass. Let’s beef it up:

    def test_returns_results(self):
        WorldEntity.next_id = 100
        world = World(10, 10)
        bot_1_id = world.add_bot(5, 5)
        bot_2_id = world.add_bot(7, 7, Direction.NORTH)
        rq =  [
            { 'entity': bot_1_id, 'verb': 'turn', 'param1': 'NORTH'},
            { 'entity': bot_1_id, 'verb': 'step'},
            { 'entity': bot_2_id, 'verb': 'step'},
            { 'entity': bot_2_id, 'verb': 'step'},
        ]
        result = world.execute(rq)
        assert len(result) == 2
        for d in result:
            print(d)
            match d['eid']:
                case 101:
                    assert d['location'] == Location(5, 4)
                case 102:
                    assert d['location'] == Location(7, 5)

That passes. I note that we have a Location instance in the returned dictionary. That’s probably going to break our JSON: it’s another connection between client and server that probably needs cleaning up. We’ll leave it for now but I’ve made a card for it.

We are green. Commit: more robust test.

Now that execute returns the results, we should be able to change DC to expect them instead of fetch them.

class DirectConnection
    def run_request(self, cohort, client_bot_id, rq):
        self.world.execute(rq)
        result_dict = self.world.fetch(client_bot_id)
        cohort.update(result_dict)

I think our stub cohort does not expect a list on update, but it should.

class Cohort:
    def __init__(self, bot):
        self._bot = bot

    def update(self, result_dict):
        self._bot._knowledge.update(result_dict)

We’ll fix that when it breaks, which it will when we do this:

    def run_request(self, cohort, client_bot_id, rq):
        results = self.world.execute(rq)
        cohort.update(results)

I said we’d be glad we had done that refactoring. We just had to change one line here. Now we have to fix our stub Cohort, because 9 tests are breaking.

    def update(self, results):
        for result_dict in results:
            self._bot._knowledge.update(result_dict)

Of course a real cohort will have to be more robust but we’re not there yet.

By the way, I can’t help noticing that our whole robot world is working just fine and we do not have a key element, a group of robots, figured out. We just have a silly little stub here. Remember what I was saying about YAGNI and all that?

We’re green. Commit: DirectConnection uses results returned from execute not fetched.

Let’s sum up. It’s Sunday and there is bacon in my future.

Summary

In a very smooth series of small steps, we now have the World returning results for all the bots mentioned in a request list, and the DirectConnection (and silly little Cohort) using those results. Although the changes were all rather small, they have made a notable improvement in how the overall system operates.

Our list of things to do looks more like this now:

  • Make it easier to create the little action dictionaries;
  • Provide and use an entry in DirectConnection to take a list of many actions;
  • Look for duplicated code in DirectConnection and take advantage of it;
  • Cause World to return all the knowledge dictionaries for all the entities in an execute list;
  • Deal with errors, including avoiding exceptions and passing error info in the knowledge.

I’ve added three notes:

  • Location in result dictionary (JSON?)
  • Refactor execute
  • Sort out adding bots

The last one refers to the fact that adding bots is handled awkwardly and in a specialized fashion. We would do well to have a way of embedding bot creation into our requests. As for the penultimate one, you can see above that execute is a bit messy looking, probably would benefit from a dose of Composed Method or something.

We have had six commits over about a 45 minute interval, not bad at all. And the program continues to improve.

See you next time!