The Repo on GitHub

I’d like to learn a bit more about creating class decorators, using decorator classes. We’ll write some more tests, and perhaps find some additional useful references. Fun! Learned something!

As I mentioned yesterday, my particular preferred practice for learning something new is to scan, skim, and peruse references and tutorials, but generally not to work through them as written. Instead, I pick up a bit of understanding, and use it to build up a program of my own, often quite unlike the material I’m examining. For me, learning a programming thing is in the doing of it, not the reading or watching, so I like to get my hand dirty as soon as seems reasonable.

I also find it valuable in that the work I’m doing tends to raise questions for me and I will often try things to find out what happens. I think that might tend to give me a bit more breadth and depth of understanding, and it probably keeps the list of things I’m wondering a bit shorter and a lot more focused.

Anyway, that’s the way I like to learn things. You may have different preferences and my guess would be that by and large, they are probably well fitted to your own learning style. You and I would probably both profit from trying something a bit outside our usual approach, just to see what happens. It’ll provide insight, and perhaps even a better idea.

Today, I’m particularly interested in two things:

  1. How might we pass arguments to our decorator? I’d like to be able to specify the exact methods to be set up for forwarding, for example, and we probably need to name the class member to be used in the forwarding. In the forwarding repo I found a few days ago, the author required parentheses on the decorator @forwarding() rather than just @forwarding, and I wonder what that’s about.

  2. How do we set up a decorator build using a class, not just a wrapper function? I think this might help with the super issue we ran into yesterday. I’m thinking that our class could contain helper methods rather than functions, and those might plug better into our new decorated class.

  3. (Wrong about the two yet again!) It seems that a class decorator can return an entirely new class rather than the one it decorates. I’d like to learn about that at some point.

We have a few tests of decorators from yesterday, so let’s start there. I think I’ll experiment before reading today. I feel lucky, and I’d like to stop feeling lucky as soon as possible.

Down To It

Here’s my second test:

    def test_deco2_class(self):
        d2 = Deco2()
        assert d2.foo() == 42
        assert d2.fargle() == 666

@deco2
class Deco2:
    def foo(self):
        return 42

The test asserts that we have a method fargle that returns 666 and yet the class does not define any such method. Here’s what makes it work:

def deco2(cls):
    setattr(cls, '__getattr__',deco2_fake.my_getattr)
    return cls

class deco2_fake:
    def my_getattr(self, key):
        if key == 'fargle':
            return lambda: 666
        else:
            return super(self.__class__, self).__getattribute__( key)

The method deco2 is of course the actual decorator and it gets ahold of the wrapped class and gives it a new attribute named __getattr__ whose value is the method my_getattr from the auxiliary class deco_fake. If I recall correctly, putting a plain top-level function in there didn’t work. Let’s check that, and document our findings with a test.

Note
In finding my way like this, I have a tendency, when something doesn’t work, to just move on and try to find something that does. It occurs to me that it might be better to write a test that documents the kind of failure one gets. We’ll try that here.
    def test_not_a_class_method(self):
        d3 = Deco3()
        assert d3.fargle == 666

And the class:

@deco3
class Deco3:
    pass

And now to write the decorator, just like deco2 except with an outside method.

def deco3(cls):
    setattr(cls, '__getattr__',open_getattr)
    return cls

def open_getattr(self, key):
    if key == 'fargle':
        return lambda: 666
    else:
        return super(self.__class__, self).__getattribute__( key)

This fails, as I suspected it would. I think we tried something very similar yesterday. The failure is:

Expected :666
Actual   :<function open_getattr.<locals>.<lambda> at 0x105f8cf40>

It seems to have returned the actual function rather than calling it.

I’d like to know the difference between the two. A test might help.

    def test_what_are_they(self):
        assert open_getattr == deco2_fake.my_getattr

The error:

Expected :<function deco2_fake.my_getattr at 0x102d163e0>
Actual   :<function open_getattr at 0x102d16520>

Huh! Those look to be the same. For some reason, Python knew to call the function that is a method of some class, and did not know to call the one that isn’t. OK, we’ve seen that it doesn’t work, but I’m going to have to do some deeper research to understand why. I remove my info test and mark the open one as @pytest.mark.xfail, indicating that we expect it to fail.

Reflection

So I have mostly learned that I don’t understand why what appears to be essentially the same code doesn’t work when the function isn’t part of a class. That could be pretty deep inside Python. More research needed.

That did take a bit of air out of my sails, but let’s do a bit of experimentation with parentheses on the decorator, and see what happens.

First I write this test, which passes:

    def test_deco2_with_parens(self):
        @deco2
        class NoMatter:
            pass
        assert NoMatter().fargle() == 666

No surprise there. Now I’ll put parens on the @deco.

>       @deco2()
E       TypeError: deco2() missing 1 required positional argument: 'cls'

OK, clearly it doesn’t want me to do that.

Discovery

I found a repo with an example that confuses me. Its decorator function doesn’t do the work and doesn’t take the class as a parameter. Instead it defines an inner function and returns that. There’s only the one commit, so I’m not sure how much to trust this repo.

A look at RealPython suggests to me that if a decorator is called without parens, the thing it decorates is passed into it. If it is called with parens, it is expected to return a function into which the thing decorated will be passed. That’s pretty magical but it would explain what we saw in that test.

I think I’ll push forward on this topic a bit. And I think I’ll start a new test file, this one is getting messy.

class TestDeco2:
    def test_deco_wrapper(self):
        @deco_p(10, 20)
        class Ignore:
            pass

        assert False

And the new decorator:

def deco_p(p1=1, p2=2):
    def wrapper(cls):
        print(f'running wrapper {cls=} {p1=} {p2=}')
        return cls

    print(f'about to return wrapper {p1=} {p2=}')
    return wrapper

We define our decorator to define a simple wrapper that prints that it is running on a class, plus the bound parameters p1 and p2. And before we return it, we print that we are about to return it, and again the parameters. We expect the failure to be preceded by the ‘about’ print followed by the ‘running’ print. And indeed:

about to return wrapper p1=10 p2=20
running wrapper cls=<class 
    'test_decorators_2.TestDeco2.test_deco_wrapper.<locals>.Ignore'> 
    p1=10 p2=20

    def test_deco_wrapper(self):
        @deco_p(10, 20)
        class Ignore:
            pass
    
>       assert False
E       assert False

This is good news!

Yes, this pleases me. I’ve consolidated the information that I read, saying that if you call a decorator with parameters, it must return a wrapper, by writing a simple decorator that does just that thing. And the test shows how that the inner wrapper function can access the parameters right out of the outer parameter list. Very nice. Doing it makes it more real to me, and, I fondly hope, gives me a much better chance of remembering it than I would have if I had just read it. In fact, I’m pretty sure that I have read it, perhaps more than once. Now, I’ve actually done it. Yay, me!

I think this is a good place to stop. I’ve learned some things, and I think the next phase will be to work with class-based decorators, which are probably a bit different in odd ways. I’d like to give this understanding a chance to gel a bit before adding confusion.

Commit: some tests to learn about decorators.

Summary

We learned:

  1. Methods added with setattr cannot be module-level functions, must already be methods on some class. I do not know why it’s that way.
  2. Decorators can have parameters, and if they do, they must return a wrapper function.

I tried just slamming parens on a decorator and it failed in a way that I didn’t understand. That one took a bit of research, and the excellent RealPython folks had an answer. My research first took me to a repo that I decided wasn’t helpful, with a quick diversion to the one that I looked at yesterday, and then maybe five or ten minutes scanning RealPython.

What brings it home for me is writing a tiny test that shows me just what has to be done, and as little more as possible. Helps me focus on the essence.

I still do not understand why a module-level function can’t serve as a method, but I’m sure that it cannot. I think that we’ll work on making decorators as classes next, because that will let us embed any methods we need right in our own class, instead of needing a fake extra class like I used in the tests so far.

And I’m wondering about the use of subclassing for our purpose here. Imagine a Bot class that calls methods like vision, which needs to be implemented as self._knowledge.vision. In our decorator, we create a subclass that uses a known name as the object to which we will forward things, and our subclass contains the __getattr__ that we use to decode the messages. Then we wouldn’t have to jam a new method into Bot instances … we’d instead return the decorated class with the necessary understanding.

Not clear to you? Me either … but I think I have a glimmering of how we might do it.

So this is fun, and I think we have at least one more day of learning to do, probably more than one. I hope you’ll come along and see what happens. See you then!