Decorators, Perhaps

The Repo on GitHub

The votes are still out regarding whether my __getattr__ forwarding is too clever or not. This morning, I’ll at least do a little learning about class decorators in Python. Language Preferences. Fun. Trouble. Learning styles.

I’ve posted a survey on Mastodon regarding yesterday’s forwarding that we wrote for Bot. The votes are not yet tallied, and there aren’t many. They’re running about 4:3 in favor of too clever. However, I got a couple of favorable comments from people and I have concluded that they are probably very smart people and therefore probably correct, and therefore it is probably OK. (What is “confirmation bias”, anyway?)

GeePaw Hill suggested giving the practice a name within the team, so that one could say “I’m gonna do the Frammis on the Whatnot class”, and everyone would know what was up. Definitely a good idea, and as soon as I think of a good name for this, I’ll promulgate it to myself as the team. I’m pretty sure we all agree on the idea anyway: we think it’s OK but we’re alert for hints that it’s too clever.

Language Preferences

I was thinking about language preferences this morning, while resisting getting up, specifically about how much I like Python, and how much GeePaw doesn’t like it. Most of my actual programming over the past three decades¹ has been in dynamically typed languages: Smalltalk, Ruby, Lua, Python. Hill, I believe, has done most of his work in statically typed languages like Java and Kotlin.

Imagine that you use some keyboard—probably not too hard to imagine—and that I use some other kind and I tell you how great it is, and you decide to get one. Immediately, it doesn’t feel right under your hands, the keys aren’t spaced right, they make an odd noise, and you seem to be making a lot more mistakes. You don’t like my keyboard!

Or, is it just that you aren’t used to my keyboard? Quite possibly, if you stick with it a while, you’ll come to like it better, and, oddly enough, if you go back to your old one, it won’t feel right.

A lot of what we like has to do with what we’re used to. And getting used to a programming language isn’t as easy as getting used to a keyboard. There’s a lot of thinking in programming, and much of our thinking is dealing with the things that the language, and the IDE, and the other tools don’t do for us … and much of our thinking is aligned with the things that they do do for us.

In the strictly typed languages, we have to declare almost everything. Some of them allow us to declare something var or val and will make inferences about the types. But all the strictly typed languages, and their IDEs, won’t let us go ahead with anything unless the types and names all line up. It’s very regimented, but you avoid a lot of errors that way.

In a dynamically typed language, the language will typically compile any syntactically correct statement, even if it happens to refer to a method that doesn’t exist. You can say employee.pax() and it will compile. It might or might not run.

There is overlap these days. Python, for example, is as dynamic as you might like. You could write a program that accepted new code via email and compiled it right into the existing code. But Python does allow you to “hint” at the types of things, and if you do, there are tools to check your program to see that things line up. And, in particular, IDEs like PyCharm will check dynamically.

We call the things that our tools do for us “affordances”. the ways in which they can be used, the things they let us do. Today’s languages and systems offer many such affordances, and we become accustomed to the ones we see in the tools we generally use.

Were you going to make a particular point?

Well, I did want to mention affordances like Python’s “dunder” methods², such as the __getattr__ that we used yesterday. In Python, if you want a particular class to act like an array or list, just about all you have to do is to implement the method __next__. (You’ll want a few others, depending on how much like a list you want the thing to be.) If you want to add two instances of Employee, you just implement __add__ and there you go.

Behind the scenes, Python is doing some really fancy and carefully designed gyrations to allow that kind of thing. Everything is dynamic. It’s quite intricate. When you say employee.pax(), Python compiles code that tries to find pax in the dictionary of methods on the instance, in case this particular employee implements pax. Then it checks the class. Then it checks the superclass. Then, if it still hasn’t found pax, it will ultimately call __getattr__. If your object implements that method, as ours does, it can do nearly anything, so long as it finally returns the sort of thing being looked for.

Smalltalk and Ruby are similar to Python. Smalltalk is probably the best-designed of the dynamic languages, in that it is the oldest, and had the luxury of some of the smartest programmers in the universe working on it. And, unlike the others, in Smalltalk, you’re modifying the actual programming environment directly. You can modify the compiler while it runs if you’re into that sort of thing.

The upshot is that I’m used to languages that work dynamically and GeePaw, perhaps not quite so much.

But there’s more than just what we’re used to: I suspect that there are aspects of our thought process, and our perceptions, that influence which languages we prefer. Before we get into a language discussion, which will generally turn into a language argument, and often a language riot, we’d be wise to reflect that a lot of “my language is better” really means “I’m more used to my language”.

Evil Example

I write this test:

    def test_if_method_hits_getattr(self):
        bot = Bot(10, 10)
        assert bot.fargle() == 42

This of course fails:

    def __getattr__(self, key):
        if key in FORWARDS:
            return getattr(self._knowledge, key)
        else:
>           return super().__getattribute__(key)
E           AttributeError: 'Bot' object has no attribute 'fargle'

So could we implement fargle in our __getattr__? I think we could.

    def __getattr__(self, key):
        if key == 'fargle':
            return lambda: 42
        if key in FORWARDS:
            return getattr(self._knowledge, key)
        else:
            return super().__getattribute__(key)

When we see ‘fargle’, we return a function that returns 42. Why does a lambda with no parameters work? I do not know. There is a whole world of magic in parameter lists. How does Python know, when it calls a method, whether it expects a self parameter, or not? I do not know. This is why we proceed very carefully when we set out to do things like this.

We’ll roll that back before something bad happens.

Note

But we’ve learned something. And raised a question as well. That wasn’t just a random firecracker tossed into the code: it was a little experiment to learn something about the area we’re working on, defining things dynamically.

Decorators

I plan to divert from working on the Robot World to learn a bit more about decorators. With the help of Bradley Schaefer, I explored a project to add a forwarding capability to a Python class. It was Very Intricate, and I do not fully understand it. I certainly couldn’t write it³.

I’d like to understand a bit more about decorators, especially class decorators. I’ll put some references to the pages I look at at the end of this article, and show you some of the tests I write to explore the topic.

I’ll make a new test file.

class TestDecorators:
    def test_hookup(self):
        assert False

Test fails. Fix it to say True. Passes. We’re wired up.

Inspired by reference #2 below, I try a simple decorator:

class TestDecorators:
    def test_deco1_class(self):
        d1 = Deco1()
        print(dir(d1))
        assert d1.bar() == 42
        assert d1.mumble() == "mumbling"

That’s supported by this:

def deco1(cls):
    setattr(cls, 'mumble', my_mumble)
    return cls

def my_mumble(self):
    return "mumbling"

@deco1
class Deco1:
    def __init__(self):
        self.foo = 42

    def bar(self):
        return self.foo

What happens here is that deco1 is called with the class Deco1 as its parameter. deco1 sets a new attribute, mumble into the class, which is the function my_mumble, which returns the string ‘mumbling’. And in the test, that method works.

In addition, PyCharm flags the line calling mumble(), warning that as far as it knows, there is no method mumble in the class Deco1.

So we know how to define a simple method in the class. Could we use this technique to define a __getattr__ that did something useful?

    def test_deco2_class(self):
        d2 = Deco2()
        assert d2.foo() == 42
        assert d2.fargle() == 666

That’s supported by this much:

def deco2(cls):
    setattr(cls, '__getattr__', my_getattr)
    return cls

def my_getattr(self, key):
    print(f'my_getattr {key=}')
    return super().__getattribute__( key)

@deco2
class Deco2:
    def foo(self):
        return 42

Sure enough when this test fails, it prints just what I had hoped it would — and then a surprise:

my_getattr key='fargle'

    def my_getattr(self, key):
        print(f'my_getattr {key=}')
>       return super().__getattribute__( key)
E       RuntimeError: super(): __class__ cell not found

I’m gratified that we got into our method with the right key. But the message about class cell, I do not understand. Pretty clearly the super() didn’t work. Let’s print self here, and in our _getattr__ in Bot, which works. We’ll see what’s different.

The bot prints things like this:

Bot self=<client.bot.Bot object at 0x1045902c0> key='location'

Our new getattr prints:

my_getattr self=<test_decorators.Deco2 object at 0x10466b350> key='fargle'

Note

My first inclination is always to assume that there’s a mistake in my code, so I look there first. I’ve met programmers who somehow assume there is something wrong with the computer when their program doesn’t work. Sometimes it is there computer, about once every year or ten.

Nothing obviously wrong. I’ll search on the web for that message. The results suggest to me that the issue is that my_getattr is defined outside of a class definition and that therefore it cannot find the value of super. I don’t disbelieve that, but I’m not sure just what to do about it.

Let’s try this:

def deco2(cls):
    setattr(cls, '__getattr__',deco2_fake.my_getattr)
    return cls

class deco2_fake:
    def my_getattr(self, key):
        print(f'my_getattr {self=} {key=}')
        return super().__getattribute__( key)

Still doesn’t work but we get a new error:

self = <test_decorators.Deco2 object at 0x106a75580>, key = 'fargle'

    def my_getattr(self, key):
        print(f'my_getattr {self=} {key=}')
>       return super().__getattribute__( key)
E       TypeError: super(type, obj): obj must be an instance or subtype of type

Unfortunately it’s not telling us what type or obj are, but it is possible that the fact that I’ve plucked the method out of one class and tucked it into another is the issue.

I found this page, which is interesting but at first glance doesn’t solve my problem.

Another page leads me to try this:

class deco2_fake:
    def my_getattr(self, key):
        print(f'my_getattr {self=} {key=}')
        return super(self.__class__, self).__getattribute__( key)

This elicits an error that I believe I can fix:

    def my_getattr(self, key):
        print(f'my_getattr {self=} {key=}')
>       return super(self.__class__, self).__getattribute__( key)
E       AttributeError: 'Deco2' object has no attribute 'fargle'

It’s true, it’s true, there is no fargle … yet.

class deco2_fake:
    def my_getattr(self, key):
        if key == 'fargle':
            return lambda: 666
        else:
            return super(self.__class__, self).__getattribute__( key)

My test passes. Enough learning for one morning, I think.

Summary

We’re two hours in and we have a thin vine slung across the chasm, at the bottom of which is a raging rapids, which if we were to fall into it would surely dash us to pieces on the jagged rocks, if the fall alone did not kill us. Before we actually try to cross this chasm, we have a lot more learning to do. But we have bridged the chasm just a bit.

Here are some things I want to understand before this study is even sort of complete:

1. How do we define a method that uses self? Our lambda takes no parameters, not even a self.
2. What is a more reasonable way to have our super method compiled properly, rather than creating a dummy class?
3. Would creating a decorator class rather than function help with #2?

What I’ve learned is that when the decorator is a function, that function is called with the class, which seems to be fully defined when our decorator is called. In our case, we “just” added additional methods to the class.

For our original purpose, some specialized accessors, it’s possible that we could define them directly rather than do that perilous _getattr__ definition. Might be interesting to try that …

The super function gets some kind of special treatment during compilation, apparently binding in some specific information about the class inside which it is called. Providing the implicit parameters allowed us to proceed, and seems to be a somewhat acceptable thing to do.

Overall, we’ve learned quite a bit about Python decorators. I’d like to comment on how we learned what we learned. It’s pretty idiosyncratic and also kind of my personal way of doing it,

I started by looking for articles that seemed to be addressing what I was trying to do. When I found one, I didn’t use its examples at all. Instead, as soon as I got a glimmer of how it might work, I started an experiment, in a test file, that was focused more directly on what I wanted to know about, providing a __getattr__ method.

I could have worked through the provided example. I could have found and watched a video. I could have found and studied RealPython’s tutorial on decorators: I’m pretty sure they have one. I didn’t do that.

I prefer to set off on my own very rapidly, with most tutorials or explanations. That might be a function of my long experience, or just a fluke of my personality and “mind”. I believe that what happens when I work that way is that my examples are more focused on what I really need. And I observe that I often read a bit, code a bit, read further, or search elsewhere, read a bit, come back and adjust my example.

Your way of learning may well be quite different from mine. There’s probably value in paying attention to the ways that work best for you. My way limits the time I spend in general learning, and since I turn to programming videos only very seldom, my video time is very limited. My focus is mostly on creating and understanding my own tests and code.

YMMV. There is more than one way.

A fun and valuable morning for me. See you next time!

“References”

1. Using Python Class Decorators. Possibly useful for writing a class that is a function decorator. Not focused on writing decorators intended to decorate classes.

2. Understanding Python Decorators: A Guide to Using Class Decorators. This one provides an example of adding logging to an entire class.

3. A valuable clue found here, amid the usual stackoverflow chaos.