The Repo on GitHub

Having abandoned the decorator thread for now, I propose to test-drive a forwarding mechanism based on descriptors. It goes quite nicely.

I expect this to go nicely. Our existing descriptor test was straightforward, and I’ve pretty much confirmed that my understanding of how they work is good enough. There’s more to learn—descriptors are quite powerful and appear elsewhere in Python—but I’m confident that we can do our forwarding properly. We’ll back up that confidence with tests, of course.

What are we trying to do?

Good question! It’s always a good idea to think about what we’re about to do before doing it. We have a class Bot that keeps most of its detailed information in a member variable, called _knowledge at present, which is little more than a record containing the information that we get back from the World server when our Bot takes action. From the Bot’s viewpoint, this is just stuff that it knows, but we keep it isolated. From Bot’s viewpoint, _knowledge is read only.

Originally, we had a lot of @property getters and setters for each element of knowledge. That made for about 40 lines of boilerplate code that made Bot difficult to read, just because of all the extra stuff. Today, Bot has specialized dunder methods __getadder__ and __setattr__ that check to see what is being accessed, and forward to the knowledge in the case of getters, and disallow the operation in the case of setters:

FORWARDS = ['direction', 'holding', 'id', 'location', 'vision']
class Bot:
    def __getattr__(self, key):
        if key in FORWARDS:
            return getattr(self._knowledge, key)
        else:
            return super().__getattribute__(key)

    def __setattr__(self, key, value):
        if key in FORWARDS:
            raise KeyError(f"cannot set _knowledge attribute '{key}'")
        else:
            super().__setattr__(key, value)

That code intercepts all attribute accesses to Bot, forwards or denies the ones it handles, and allows the rest to do what they will, by calling super(). It’s OK, but rather deep in the bag of tricks. I think what we do today will be better. Let’s get started.

Test-Driving Forwarding

I begin with this test:

    def test_forwarding(self):
        needy = NeedsInfo()
        assert needy.info == "info"

The idea here is that the NeedsInfo class uses an auxiliary class to provide its info. The test runs with these two class definitions in place:

class InfoHolder:
    def __init__(self):
        self.info = "info"

class NeedsInfo:
    def __init__(self):
        self.holder = InfoHolder()

    @property
    def info(self):
        return self.holder.info

We see that NeedsInfo creates an InfoHolder() and then forwards the property info to it. The test is green. Let’s commit it just because it’s green: initial test for forwarding.

But that’s not the definition we want for NeedsInfo. This is:

class NeedsInfo:
    info = Forwarder('holder')
    
    def __init__(self):
        self.holder = InfoHolder()

What we mean by this, of course, is that the info of NeedsInfo is to be forwarded to its member named ‘holder’. We’ll use our new knowledge of descriptors to write the Forwarder class.

This was almost too big a bite, and in fact I went in two steps, which I’ll describe below. But this object passes our test:

class Forwarder:
    def __init__(self, forward_to):
        self.to = forward_to

    def __set_name__(self, owner, name):
        self.name = name

    def __get__(self, instance, type=None):
        forward_to = getattr(instance, self.to)
        return getattr(forward_to, self.name)
Two Steps
I went in two steps to create the get. First time through, I coded getattr(self.holder,self.name. That meant that I only had one possible incorrect getattr to deal with. When that worked, I wrote the forward_to line.

What would have been even better would be to Extract Variable to create the forward_to line and then edit it, but I didn’t think of that.

I’m not fond of the names, but explaining it here will help with that.

  1. The Forwarder is created with the name of the attribute to which it is to forward its messages.

  2. When the Forwarder is used to create the info variable in NeedsInfo, the __set_name__ is called, passing the name ‘info’, the name of the variable to which Forwarder is being assigned. Now the Forwarder instance knows both the name of the object it should forward to, and the name of the attribute it is supposed to fetch.

  3. When the test accesses needy.info, the __get__ method is called. It fetches forward_to, the attribute of needy that contains the information we need, and then, using that object, fetches the attribute named ‘info’.

We have not yet dealt with setting the values. In our particular case, we want it to be impossible to set them. We’ll get there, but first let’s rename a bit. What is a good name for the object to which we forward messages? I do not know. “forwardee” doesn’t seem right. How about receiver? I think I like that. Rename a bit:

    def test_forwarding(self):
        needy = NeedsInfo()
        assert needy.info == "info from InfoHolder"

class Forwarder:
    def __init__(self, receiver_name):
        self.receiver_name = receiver_name

    def __set_name__(self, owner, attribute_name):
        self.attribute_name = attribute_name

    def __get__(self, instance, type=None):
        receiver = getattr(instance, self.receiver_name)
        return getattr(receiver, self.attribute_name)

class InfoHolder:
    def __init__(self):
        self.info = "info from InfoHolder"

class NeedsInfo:
    info = Forwarder('extra_data')

    def __init__(self):
        self.extra_data = InfoHolder()

OK, that’s better. Now we need an additional test raising an exception if we try to set our info value.

    def test_cannot_store(self):
        needy = NeedsInfo()
        with pytest.raises(AttributeError):
            needy.info = "cannot do this"

This fails, of course, but we can fix that right up:

class Forwarder:
    def __set__(self, instance, value):
        raise AttributeError(f"cannot set '{self.attribute_name}'")

And we are green. I extend the test to check the message:

    def test_cannot_store(self):
        needy = NeedsInfo()
        with pytest.raises(AttributeError) as error:
            needy.info = "cannot do this"
        assert str(error.value) == "cannot set 'info'"

Green. Commit: forwarding tests passing.

I think that our Forwarder class is exactly what we need in Bot, so let’s move it to the client folder. PyCharm is happy to do that for us.

Tests remain green. Commit: Move Forwarder class to client folder.

Reflection

I want to jump right on installing this baby into Bot but it seems prudent to think for a moment before leaping in. Here’s the Forwarder again:

class Forwarder:
    def __init__(self, receiver_name):
        self.receiver_name = receiver_name

    def __set_name__(self, owner, attribute_name):
        self.attribute_name = attribute_name

    def __get__(self, instance, type=None):
        receiver = getattr(instance, self.receiver_name)
        return getattr(receiver, self.attribute_name)

    def __set__(self, instance, value):
        raise AttributeError(f"cannot set '{self.attribute_name}'")
  • We init with the name of the member containing the attributes we want.
  • We set the attribute name for our particular instance.
  • On set, we raise an error.
  • On get, we get the proper attribute from the caller instance and then get the desired attribute from that via its name.

Seems right to me. I think we can change Bot like this:

class Bot:
    direction = Forwarder('_knowledge')
    holding = Forwarder('_knowledge')
    id = Forwarder('_knowledge')
    location = Forwarder('_knowledge')
    vision = Forwarder('_knowledge')

Irritatingly, one test fails. What is it?

    def test_cannot_set_into_knowledge(self):
        bot = Bot(10, 10)
        with pytest.raises(KeyError):
            bot.id = 101

Outstanding! We changed to an AttributeError. Improve that test:

    def test_cannot_set_into_knowledge(self):
        bot = Bot(10, 10)
        with pytest.raises(AttributeError) as error:
            bot.id = 101
        assert str(error.value) == "cannot set 'id'"

Green. I’ve removed the __getattr__ and __setattr__ from Bot, of course. Commit: Bot now uses Forwarder instead of custom-made get and set attr.

Summary

Bot used to have two gnarly methods and a special list:

FORWARDS = ['direction', 'holding', 'id', 'location', 'vision']
class Bot:
    def __getattr__(self, key):
        if key in FORWARDS:
            return getattr(self._knowledge, key)
        else:
            return super().__getattribute__(key)

    def __setattr__(self, key, value):
        if key in FORWARDS:
            raise KeyError(f"cannot set _knowledge attribute '{key}'")
        else:
            super().__setattr__(key, value)

This code provides for property-style accessors for those five names, fetching the values from the _knowledge attribute. It works: it’s just that it is gnarly and very custom made for this situation.

Our new improved code in Bot is just this:

class Bot:
    direction = Forwarder('_knowledge')
    holding = Forwarder('_knowledge')
    id = Forwarder('_knowledge')
    location = Forwarder('_knowledge')
    vision = Forwarder('_knowledge')

More compact, more clear what it does, keeps Bot clean, isolates the weirdness to Forwarder, and since forwarder is a very vanilla descriptor, it’s not as deep in the bag of tricks as we were. And I’m proud of myself for test-driving it. Little pat on the back for me there.

I find this pleasing. I’m not sorry that I learned so much about decorators: it’s good to know things. And I’m happy that my decorator research gave me three gifts, knowledge about decorators, knowledge about why we can’t just casually write new decorators, and a pointer to descriptors.

We’ve wandered a bit, picking up knowledge as we went, and we’ve wound up in a better place.

Quite pleasing. See you next time!