Python Asteroids+Invaders on GitHub

Today, in the spirit of meta, I work on a metaclass for ignoring methods. I start on Wednesday and finish on Thursday. It’s scary weird.

I am back to test-drive a spike for metaclass. The plan is this:

Provide a metaclass that accepts a list of method names to “ignore” and defines them all as pass in the class dictionary. Arrange things such that if the class subsequently defines one of those ignored methods, it will work. Try to avoid weird messages from Python or PyCharm. Deploy this only if it feels right.

I’ll test-drive it, and start with the metaclass class in the test, moving it to prod only if it seems right.

Much Trouble Ensues

I worked a while trying to create a metaclass to no avail. So I solved a smaller related problem. That, too, took a fair amount of fiddling. I did not record my usual every move: we’ll look at the resulting tests and code. I warn you, this is pretty rough: it’s scratch work, aimed at learning, and was built over many small steps. Here’s the test:

    def test_exists(self):
        foo = Foo()
        assert foo.abc(2) == 42
        foo.interact_with_bar(33, [])
        with pytest.raises(NotImplementedError):
            foo.interact_with_baz(33, [])
        fleets = []
        foo.interact_with_qux(33, fleets)
        assert fleets, "erroneously ignored"
        assert fleets[0] == 42

We see that our test class Foo includes an abc method, and that our test expects it to have interact_with_bar, and interact_with_qux, but that interact_with_baz should raise an exception. Finally, we see that interact_with_qux is supposed to have put a 42 into the fleets parameter.

These are all steps along the way to making it work. The last bit emulates what we’ll need in the real version, overriding interact_with_* methods.

Here is class Foo:

class Foo(Ignorables):
    ignore_list = ["interact_with_bar", "interact_with_qux"]

    def __init__(self):
        super().__init__(self.ignore_list)

    def abc(self, value):
        return 40 + value

    def interact_with_qux(self, thing, fleets):
        fleets.append(42)

The easy bits first: it implements abc and interact_with_qux, and we can see immediately how they work to pass the test. What about that ignore_list? It asks to ignore interact_with_bar. If that works, our test will not fail on the call to interact_with_bar. The test runs, so we know that method was ignored, though we have not yet seen how it was done.

We also see that it asks to ignore interact_with_qux. If that happens, our test will break, because our implementation above will not be found. But the test does run, so we know that defining the method overrides ignoring it. That’s in our desires above.

How does this happen? Like this:

class Ignorables():
    def __init__(self, ignores=[]):
        for ign in ignores:
            if not ign in self.__class__.__dict__:
                setattr(self, ign, self.none)

    def none(self, o, f):
        pass

    def interact_with_bar(self, _object, _fleets):
        raise NotImplementedError

    def interact_with_baz(self, _object, _fleets):
        raise NotImplementedError

    def interact_with_qux(self, _object, _fleets):
        raise NotImplementedError

The superclass of Foo is Ignorables. Foo calls super()__init__ with its list of names to ignore. The __init__ in Ignorables grabs the class dictionary and if the name isn’t already there, puts it in, pointing to the method none, which has the right signature and does nothing.

So, when this is done, the Foo class dictionary will contain implementations for interact_with_bar and interact_with_qux, ensuring that the not implemented errors will not occur for those. Since baz was not mentioned, it is still subject to the NotImplementedError.

All this works. It’s already pretty deep in the bag of tricks: jamming methods into the class dictionary is really best left to the compiler, but it does work.

We’ll assess how deep this all is when we’re done.

At this point on Wednesday afternoon, I knew one way to stuff methods into a class dictionary such that they would override the superclass’s NotImplemented versions, and would be overridden by real methods defined in the class.

That’s essentially what we’re looking for operationally, but we don’t want to do it this way if we can do it better with a metaclass. At this point, Wednesday afternoon, I did not know how to do that part. But tomorrow will have been another day.

Starting Thursday AM

After a lot of fiddling, I have something working. It’s quite intricate. I shall try to explain it.

The class Base is the superclass of all the classes we build. It is analogous to the abstract class InvadersFlyer, but it is no longer abstract: it implements all the needed methods as errors. Why? Because I don’t see how to work with abstract methods while using this scheme. We’re essentially inventing a new kind of abstract method scheme.

Base subclasses must implement foo, bar, baz, and qux or they will get NotImplementedError at run time.

class Base():
    def foo(self):
        raise NotImplementedError

    def bar(self):
        raise NotImplementedError

    def baz(self):
        raise NotImplementedError

    def qux(self):
        raise NotImplementedError

We propose to implement a class FooIgnores that wants to ignore the bar and baz methods … but the programmer has made a later decision to implement baz after all. When this happens we want bar to still be ignored, but baz to work as defined.

Note
I see in post that FooIgnores looks like Fool gnores but in fact it is Foo Ignores with a capital I. Thanks, font. I’ll tick-mark them.

Classes with the ability to ignore methods are defined like this, with a reference to a metaclass and a list of method names to ignore:

class FooIgnores(Base, metaclass=IgnoreThese, ignore=["bar", "baz"]):
    def foo(self):
        return "foo"

    def baz(self):
        return "baz"

To create your class so that it ignores bar and baz, you use the metaclass IgnoreThese, and list the names of the methods you want to ignore in the ignore list parameter.

Now here is the test:

    def test_create(self):
        foo = `FooIgnores`()
        assert foo.foo() == "foo"
        assert foo.bar() is None
        assert foo.baz() == "baz"
        with pytest.raises(NotImplementedError):
            foo.qux()

We implement foo in FooIgnores, so it works. We asked for bar to be ignored and it returns None, showing that it was properly ignored: otherwise it would have raised NotImplementedError.

We implemented baz, even though we asked to ignore it, and our implementation works, returning “baz”.

We neither implemented qux nor ignored it, so when we call it, we get NotImplementedError as expected.

So this is working. What is the metaclass IgnoreThese doing?

class IgnoreThese(type):

    @classmethod
    def __prepare__(metacls, name, bases, ignore=None):
        result = {}
        if ignore:
            for name in ignore:
                result[name] = lambda x: None
        return result

    def __new__(cls, name, bases, classdict, ignore=None):
        result = type.__new__(cls, name, bases, dict(classdict))
        return result

I can only explain this somewhat: it is the result of a lot of reading and a desperation shot at the buzzer.

In a metaclass, the __prepare__ method’s job is to return a dictionary-like object that is to be used as the class dictionary for the class. In our case, for each name in the ignore parameter, we make an entry with that name and a one-parameter lambda returning None. The one parameter will be self in the calls.

So the class dictionary is returned with pre-defined None-returning methods for each name provided, in our test case, bar and baz.

The documentation does not make this clear but it seems that we must define __new__ as well. I am guessing that classdict is the dictionary-like thing that we returned from __prepare__. I think it’s a good guess. The result is just boilerplate that I stole from the documentation. What it does, I guess, is what you always do, if you are down in the guts of things, to define a new type.

I don’t really understand why that __new__ is necessary. Without the __new__ you get an obscure error about __init_subclass__ not expecting keyword arguments. I suspect it’s needed to provide for the keyword argument, but I was unable to really figure out the documentation. I copied the example, it worked, I tweaked it a tiny bit, and then I backed away slowly.

Let’s reflect and sum up for yesterday and this morning.

Reflection

The whole process of getting here was so chaotic that I didn’t even try to write it up as I went along. Partly it was just that I was experimenting, but partly it was so much try this, doesn’t work, try that, doesn’t work, try other thing, still doesn’t work, that the article would have been hell to read, yes, even more than my usual hellish articles.

But what I always had in mind was a vision a bit like this:

Put a null method of some kind in the object’s dictionary, based on a list of method names that should get the null method. Do it in such a way that if something is in the list but also defined explicitly as a method, the explicit definition is used, not the null method.

I knew that much because I sort of know how OO systems work, and I sort of know that Python keeps attributes in dictionaries. I was not thinking about the difference between the class dictionary and the object’s own dictionary, but because of long experience with these things, I was ready for the idea when my nose was rubbed in it.

So I kept working to put null methods into dictionaries. The first tests, yesterday, put them into the object dictionary. The second test, today, addressed the metaclass issue and putting the null methods into the class dictionary instead.

So cut and try, with many many very small cuts, most of them missing arteries, I inched toward things that worked, extended my test, inched forward.

The process was frustrating when I tried the metaclass first, and I really made no progress, probably because of the need for the __new__, which I only ran across this morning. Fortunately, I backed off from metaclass pretty quickly and tried the first test shown here, which does the trick in the object dictionary, not the class dictionary.

By the time I had that test running. I was tired and stopped for the afternoon. I had other distractions that consumed energy as well, but it was probably good to stop in any case.

Then, this morning, I was fresh as a daisy at 0630 and started again. I knew that to create the test I wanted, I would need a test class, a base class, a metaclass, and a test subject class. I felt that the process of describing building those out was beyond me and decided to do as I have done, make it work and then report.

I say “beyond me”. What I really mean is that when I program and write an article at the same time, my mental effort is divided between the programming and the writing, and I am continually switching modes from one to the other. I felt that today’s thing deserved all the concentration I could provide and that writing contemporaneously was going to get in the way even if I didn’t try to spell “contemporaneously”.

And it went smoothly this morning until I started getting that weird error, which was breaking pytest. Pytest recompiles your classes, so if they won’t compile, pytest itself breaks. And it doesn’t produce the console output, so prints in there do not help. But after a bit of internetting, I found that example with the __new__ and pasted a copy in. That error went away and I got another, which led me to the current version fairly directly.

So aside from “a miracle occurs and I pull __new__ out of some orifice”, today went smoothly as well.

Summary

My metaclass works and overrides methods so as to hide the superclass’s fatal error versions, and allows the class being created to implement methods that override even the null overrides. That prevents what I think will be a probable error, where we set up a class to ignore some interaction methods, then later decide to implement one, and we forget to remove it from the override list. The new method will prevail. (We will not get a warning: I’m not sure how we could do that if we need to, but it fails safely.)

So I have what I wanted, the ability to provide a compact list of interact_with_* methods that my class does not want to implement, and have those nulled out, while still allowing for errors to be detected if we neither ignore nor implement a required method. And I think that it’s pretty close to the right way to have done it.

Your scientists were so preoccupied with whether or not they could, they didn’t stop to think if they should.
– Dr Ian Malcolm

Right, thanks, Ian. Now we can do this. Should we do it?

Pro
Code today is cluttered with empty methods. In some cases, half of a class is empty methods. This scheme would reduce that to a simple list of names in the class definition.

The question of what a class pays attention to would be somewhat easier to work out, both because the class will be smaller, and because if the method does not appear in the ignore list, the class must process it.

In short, our code will be easier to write, shorter, and easier to understand.

It seems very unlikely to fail. We would welcome more tests to try to make it fail.

Even if it did fail, we do not see a way in which it could fail that would result in a defect in the program. It could make the program fail to compile, but we don’t see much chance of a covert error.

Con
This is quite deep in the bag of tricks. It is tested well enough that we can be sure that it works, but no one on the team really understands metaclasses well enough to fully understand this.

It’s clever. Clever code should be avoided.

It’s weird and scary and scary weird.

I think that sums up the pros and cons pretty well. I’m sure you know what I’m going to do next.

See you next time! Bring dinosaur repellent.