The Repo on GitHub

I found a very strong series of articles about Decorators. They make a good case that I’m doing them wrong. What do we think about that? We change direction.

The article series is How you implemented your Python decorator is wrong. The fundamental starting point of the series is that “most people who write decorators” write them such that the scope in which they can be used is quite limited. The author, Graham Dumpleton, makes his case quite well, and even provides a package, ‘wrapt’ that does decorators right by his lights. I would be proud to have done such a good job on anything.

Dumpleton’s series sent me on a bit of a searching frenzy last night, where I learned just a bit about all the magical things that are lurking beneath the surface of Python. Since my learning of Python has come almost entirely from reading just enough to write the little programs that I like to write, just enough to show how I might write code and tests, there are things that perhaps every real Python programmer knows, that I do not.

I didn’t know that you can call help on any Python object and get a textual report about it.

Help on decorated in module test_decorator_classes object:

class decorated(Decorated)
 |  decorated(*args, **kwargs)
 |
 |  Method resolution order:
 |      decorated
 |      Decorated
 |      builtins.object
 |
 |  Methods defined here:
 |
 |  __init__(self, *args, **kwargs)
 |
 |  bar(self)
 |
 |  ----------------------------------------------------------------------
 |  Data descriptors inherited from Decorated:
 |
 |  __dict__
 |      dictionary for instance variables
 |
 |  __weakref__
 |      list of weak references to the object

I was vaguely aware of the __doc__ information being available, but since I generally avoid doc comments, I didn’t think about how a wrapper around a class might need to do something special to make __doc__ work right. And so on.

Dumpleton’s project, ‘wrapt’, provides what he considers to be the proper way to create decorators, and more. At this writing, he is offering me far more information than I am prepared to study and adopt. If I were trying to create perfect decorators, I would study what he has done quite carefully, and if you’re in the decorator business, I commend his work to you.

I gather that the functools.update_wrapper and functools.wrapper are pretty good, though I believe that Dumpleton has some concerns about them. Apparently the functools functions update the wrapping function’s metadata to match the metadata of the wrapped function, so that it will provide the help and docs and such of the original function. This means that, while I’ve been working without them so far, I should probably adopt them if I plan to write any even somewhat serious decorators. Of course, for now, I’m just writing some tests, in order to learn.

However,

I did have in mind trying to write a decorator that managed forwarding from the Bot to its Knowledge attribute. We presently do that with a hand-crafted __getattr__ method, which does the job and allowed us to remove a raft of properties from the class.

I think I’ll abandon that idea, in part because I’ve been slapped down by Dumpleton’s excellent work, but also because I’ve got a new scent in my nose. I was skipping around, searching and reading about decorators, in the light of my start at Dumpleton’s articles, when I ran across—wait for it—descriptors.

What little I understand here comes mostly from this RealPython article.

I haven’t written any tests or code involving descriptors yet, so this is preliminary and subject to change, but it seems that there are these things called descriptors, and in particular “data descriptors”, when associated with an object, are first in the chain of actions taken to look up an attribute of an object.

An example in the above article creates a class whose attributes can contain only even numbers, such that if you try to set them with an odd number, they receive zero instead. Clearly, if you can do that, you can do a lot of significant and useful things as well. I do not fully understand how the example works, but clearly it relies on the fact that descriptors are found at the beginning of the attribute lookup sequence, since, in the example, the attributes appear to be being defined as class variables. That puts them at the beginning of the lookup, and after that, they seem to take control of things.

I think I’ll write about their example sometime soon, because it has an interesting aspect that surprised me.

But what about decorators?

Yes, well. I am very tempted to abandon the current thread on decorators, except that, while I think I understand how to do parameterized decorators using functions, I do not see how to do a parameterized decorator that decorates a class (not a function) and that is itself implemented as a class. Our test of that runs … but only because I hacked the decorated class to accept an unexpected parameter:

    def test_parameterized_decorator(self):
        @deco_class_parm(37)
        class Decorated:
            def __init__(self, unexpected):
                print(f'Decorated init {self}\n{unexpected=}')
                self.foo = 42

        d = Decorated()
        assert d.foo == 42
        assert d.bar() == "mumble"

That’s supported by this decorator:

class deco_class_parm:
    def __init__(self, param):
        self.param = param

    def __call__(self, callable):
        class decorated(callable):
            def __init__(self, *args, **kwargs):
                super().__init__(self)

            def bar(self):
                return 'mumble'

        return decorated

I believe that the issue is that our __call__ should be returning a wrapper, not the class. But I have not found a working example yet, and I’m not quite sure what to do. I could just def a wrapper containing the class, I suppose, but Graham Dumpleton has taken much of the wind from my sails. In addition, it is becoming clear that a proper decorator for defining forwarding methods is going to be Way Too Clever to be used in our program. I wouldn’t mind knowing how to do it, but I don’t think it’s reasonable given the overall capabilities of my team, which consists of me, and you can see that my capabilities do not extend quite this far.

The descriptor approach seems more tasty, and not just because I don’t know much about it yet. What I do know makes me pretty sure that it’s just plain simpler. If that’s true, it’s a better candidate for the forwarding notion than a decorator would be. It will have far less mechanism, no double-depth wrappers. Simpler. And simpler is better.

Decision Time

I think what we’ll do is belay the decorator experimentation, at least for now, and try some descriptor tests and code. There’s nothing depending on decorators, I just wanted to learn about them, and I’m kind of chock full of some combination of learning, confusion, and frustration on the subject. So let’s turn our attention to descriptors.

Begin with a test file. My hookup works. I’m going to crib from the RealPython article for my initial test, modified a bit but all credit goes to them:

class TestDescriptors:
    def test_realpython_idea(self):
        class OnlyEvens:
            value1 = EvenNumber()
            value2 = EvenNumber()

        oe = OnlyEvens()
        oe.value1 = 4
        oe.value2 = 3
        assert oe.value1 == 4
        assert oe.value2 == 0

Skipping down to the assertion area, we see that we have attempted to set oe.value1 to 4, and oe.value2 to 3. We see by the assertions that the 4 was accepted, and the 3 was not, leaving oe.value2 set to zero.

Here’s the EvenNumber class, which I left out of the listing above:

class TestDescriptors:
    def test_realpython_idea(self):
        class EvenNumber:
            def __set_name__(self, owner, name):
                self.name = name

            def __get__(self, instance, type=None):
                return instance.__dict__.get(self.name) or 0

            def __set__(self, instance, value):
                instance.__dict__[self.name] = \
                    (value if value % 2 == 0 else 0)

I believe that I understand the above, with one caveat, so let me explain what’s going on, which will cement the ideas in my mind. For weak values of cement.

The class OnlyEvens is intended to have two attributes, value1 and value2, which are to contain only even numbers, or zero if anyone tries to put an odd number into them. The positioning of those definitions as if they were class variables is odd.

Let’s pretend that we had this class:

class DefaultValues:
    value1 = 5
    value2 = 71

If we were to create an instance of this class … well let me show you in a test:

    def test_default_values(self):
        class DefaultValues:
            value1 = 5
            value2 = 71

        dv = DefaultValues()
        assert dv.value1 == 5
        assert dv.value2 == 71
        print(f'before: {dv.__dict__}')
        dv.value1 = 6
        assert dv.value1 == 6
        assert dv.value2 == 71
        print(f'after:  {dv.__dict__}')

The prints show this:

before: {}
after:  {'value1': 6}

Before the assignment, oe has no attributes in its dictionary. Accesses to value1 or value2 refer to the class variables. After the assignment, oe has its own value1.

So, regular class variables are accessed first in the lookup chain, and when you attempt to store into one, Python creates a new attribute in the object’s dictionary at that time.

Now in our EvenNumbers class above, the situation is similar, up to a point. The class variables value1 and value2 will get first crack at the get and set. But the EvenNumbers behavior for get is to return the instance’s value, if it is present, and otherwise zero. And its behavior for set is to check the value for being even, substituting zero if it is not even, and to store that in the instance value.

Since the class variable always gets the first chance at an access, it will continue to apply that behavior indefinitely.

I mentioned a caveat: I do not understand is what the type parameter is. I’ll add a print to see what it is. The result is:

type=<class 'test_descriptors.TestDescriptors.test_realpython_idea.<locals>.OnlyEvens'>

That appears to be the class in which our EvenNumbers instances are being used. I’ll see if more reading tells me what use one might make of that information: I don’t see any immediate use that I’d have for it.

Summary

I think we’ve reached a stable point this morning. We have a test similar to the RealPython example, and we can explain what it does and how it does it, though we see that it has access to information that we have no present use for. And we have a little auxiliary test of class variables that helped us understand what’s going on.

Since the instances of these descriptors get the name of the variable, it seems to me that we could have a little descriptor, Forwarder, and we could write something like this:

class Bot:
    direction = Forwarder('_knowledge')
    holding = Forwarder('_knowledge')
    id = Forwarder('_knowledge')
    location = Forwarder('_knowledge')
    vision = Forwarder('_knowledge')
    ...

And all those names would behave like attributes, forwarding to the _knowledge attribute. We could even have a parameter to say whether the forwarder should allow setting, if we need that.

Certainly I don’t consider that solved, but I feel quite sure that we could do it, that it would be easy, and I am sure it would be far less clever and far more clear than a decorator. It’s probably better than our current overriding of __getattr__ and __setattr__ as well.

I feel good about this. I think following the descriptor path will lead to something much better than digging more and more deeply into decorators. I’m glad that I discovered Dumpleton’s articles, and I think it was wise to belay the decorator exercise and delve into descriptors.

I hope you agree. See you next time!