P-265 - But No

Now that we have this metaclass, and now that we have this decorator, what shall we do? The answer may surprise you.

Testing to Discover and Learn

My study of decorators resulted in P-264 - Decorator Tutorial. With that in place, we’ll skip over the discovery tests that I wrote so as to build up an understanding of class-based decorators. The approach is simple, though of course it is always a different flow depending on what aspect of which system I’m trying to learn.

To learn some new thing, some folks will work at the command line, typing things in and observing what happens. I find that tedious and it’s hard to remember all the things that pass by. Some folks will write a little main program and evolve that until they get something working. I might do that as well, but such a thing is inherently meant to be thrown away.

In recent years, I’ve been developing the habit of writing tests when I am learning some new language feature or API. I get several advantages from that.

When I’m done, I have a permanent record of what I’ve learned, with tests right in the very program I was really working on. (There is a drawback to that, I suppose, in that the tests aren’t in some public library when I can find them even on next year’s project, but I don’t have such a repository and I’ve never seen one really work well. Anyway I don’t do that.)
I can write a test and repeat it easily until I learn what it’s intended to teach. Repeating tests in the REPL is much more tedious. Tests are at least as easy to repeat as rerunning the little main program.
When a test yields the information I need, I can enhance it, going to the next step, or I can save that test, ending with an assertion about whatever I’ve learned, and create a new one. This second way produces an interesting learning history. Not so much for going back a year from now: more for going back ten minutes from now to check what that previous test really did.

So this scheme works best when I create a series of tests, each one delving a bit further into whatever I’m trying to learn. The Decorator Tutorial basically packages up my final tests, making them a bit nicer for publication.

Recent Learnings

I’ve followed that scheme more or less well, over the recent articles in this series, starting back in P-257 - metaclass, up through today. I’ve learned how to build a metaclass and use it to create special definitions of methods that my objects want to ignore. Then I’ve learned how to build Python decorators, which can do that job without digging quite so deeply into the bag of tricks.

I am happy to learn both of those things. I like knowing things like that, and I often find that having a deeper understanding of the language and system I’m working with pays off. The question before us today is:

Now that we know a fair bit about metaclasses and decorators, what are we going to do that we prefer over the current abstract class scheme.

Let’s review.

Tensions, Forces

Things are as they are because they got that way, Jerry Weinberg told us, and in this program, we have a very simple hierarchy of objects, from Flyer, to InvadersFlyer and AsteroidFlyer, with the game objects deriving from one or the other of the latter two.

There is some use of implementation inheritance in those superclasses, and some us of abstract methods as well. The difference between the two is substantial:

Abstract Methods: The abstract methods require each subclass to implement all the existing abstract methods, even if they give it an empty implementation with pass. This gives us a bit of confidence that there was at least a moment of consideration as to how each object interacts with each other object. That confidence should be pretty low, because we can implement missing methods with PyCharm, and it just sets them all to pass. When we implement a new Flyer, ThatGuy, we get a new double-dispatch method interact_with_thatguy, and if we make that method abstract, all the other subclasses have to implement it, or the system won’t run. This way is tedious, and a bit more secure.
Implementation Inheritance: We can implement interact_with_thatguy as non-abstract, with pass in the superclass. That saves us from needing to make changes to the subclasses to keep the system running. But we lose the confidence that the abstract method would bring us, that at least for a moment, we had to consider each class and its relationship with ThatGuy. This is not just a concern: it has actually happened, at least once, that we should have implemented the new method somewhere but mistakenly left it set to pass. But this way is really much easier, with much less editing.

The Abstract Method approach is more pure, and far more tedious. The Implementation Inheritance approach is much easier, considered by many to be impure, and it has a small but definite negative impact on quality.

The code is longer and a bit harder to read with Abstract Methods, shorter and thus easier to read with Implementation Inheritance.

The “Ignore” Idea: It came to me that if a class could declare, in some simple way, a list of methods that it ignores, there would be an explicit statement required, and therefore an equivalent level of confidence to the Abstract Method approach, but without the noise in the code caused by all those three-line almost identical pass methods.; So we have built two ways to provide that declaration, using metaclasses and decorators. With just a line or two, we can mark any class as ignoring as many of the interact_with_thatguy methods as we wish.

The question is, what shall we do? The answer may surprise you.

Abstract Methods FTW

I’m going to go to full use of abstract methods. Even I am kind of wondering why. Here’s my thinking, to the extent that I know it:

Abstract Methods give the most confidence we can have, and are checked at compile time, not run time.
Implementation Inheritance is legitimate, in my view, but it is also always a bit surprising. We’re like hey, what happened, oh, it got dealt with in the superclass. I want to avoid writing code that surprises me later.
The savings in code clarity from the metaclass or decorator approach is minimal, and using those approaches shifts error detection from compile time to run time.
The “ignore” feature does not avoid the need to edit all the subclasses. If we add a new interact_with_thatguy, we have to add it to everyone’s ignore list. (We could possibly consolidate those lists somewhere, but honestly they belong with the class they decorate.)

So the new rule is that all those interact_with_thatguy methods are to be made abstract, not to be inherited, and to stop using implementation inheritance for that purpose.

There are two prices to pay. There is the immediate large price where we implement all those methods as abstract and then have to edit a lot of Flyer subclasses. And there is the smaller ongoing price, because every time we create a new Flyer subclass, we’re going to have to do a small edit in a growing number of other classes.

This change is going to slow us down, not much, but a little bit. I do not like that, but I think that the metaclass and decorator schemes are too deep in the bag, and they do not save us from the slowdown. Implementation Inheritance is the fastest, but it is also more risky.

In the absence of a huge savings in time, I think sticking with the formally ideal approach of abstract methods is the thing to do.

Now let’s change one and see how awful it really is.

Switching to Abstract (Time: 0750)

I plan to begin by reordering the methods in InvadersFlyer to be in alphabetic order without regard to whether they are marked abstract.

Having done that (0752) I decide I’ll just mark them all abstract and do it in one go. I can always roll back if I need to.

Once I do that, 20 tests fail and 75 (at least) are not even started. (0755)

I think PyCharm’s code inspection may be useful here. Analyze Code lists all observed issues including showing all the classes that no longer implement required abstract methods. PyCharm considers that to be a “weak warning”, but for us it is far more serious, I will just implement them via PyCharm, which will put them in a nasty place in the file but we’ll get back to green quickly. I believe.

Asteroid class done (0759)

All done (0806). One test failing, probably that checker one.

>       begin = BeginChecker()
E       TypeError: Can't instantiate abstract class BeginChecker with abstract methods interact_with_fragment, interact_with_gameover, interact_with_saucermaker, interact_with_score, interact_with_scorekeeper, interact_with_shipmaker, interact_with_signal, interact_with_thumper, interact_with_wavemaker

I thought I could get away without adding all the methods to the checkers, but no. Do it.

We are green (0808) Commit: Converted all interact_with_ to abstract, implemented as pass in all subclasses. Green*

A 15 class commit (0809).

So twenty minutes for a slapdash job that puts all the passed abstracts at the top of the class, because that’s where PyCharm puts them. I also removed the metaclass from Bumper.

We will clean up the individual classes, reordering the methods, as we pass through them. That is the general rule, by the way: I think it’s preferably, on a real project, to leave as much refactoring to be done when we are working on a story in the area needing improvement. That helps to ensure that the changes we make tend to pay off in improved speed. It only slows us down to refactor code that will never be looked at again,

Refactoring is kind of fun, I’ll admit, and I do it all the time here, but that’s because I want to show that incremental refactoring works well. In a business sense, improving code that isn’t going to change doesn’t buy us much, if anything.

Now, if you’ll permit, I’ll take a little break.

Reflection

I kind of feel like I’ve taken the high road here, but not without some pain. Given the alternatives before us, I feel that the abstract method approach is the one that the conventional wisdom would have us use. But it is more than a little bit inconvenient. I think we can be pretty confident that every time we add a new object to the mix, we are in for about twenty minutes of adding a new method to all the subclasses. That effort is just about completely waste. We may possibly take the occasion to think about each individual class and how it should interact with the new one, but my life experience suggests that I’m more likely to just crank through blindly adding pass methods to everyone. I’m unlikely even to reap the small benefit of improved insight.

However, until an obviously better idea comes along, I think it’s best to stick with the tried-and-true abstract method approach, with the implementation inheritance one a very close second. How close? On some days, I’d prefer it, that’s how close.

When we find ourselves with no great alternatives, we often do well to settle for the least bad from the list, and I think about 2/3 of the time, that abstract methods are least bad. But I think we also do well to continue to think about alternatives. We have benefited from the metaclass experiment and the decorator experiment. The decorator, in particular, is quite likely to come in handy at some future time. As for the metaclass, that’s very deep in the bag of tricks and I would be wise to leave it there until I learn more than I now know.

Other Ideas?

But what about other models, other ways of classifying our objects? Right now, all the objects used in Asteroids are under AsteroidFlyer and all the objects in Space Invaders are under InvadersFlyer. But it seems to me that there are some different classifications that we could make, but do not make presently.

Let me make up some terms. I expect the terms to change, but they may help me make useful distinctions.

Active Interacting Objects: The ships, asteroids, invaders, guns, saucers, and shields area all part of the game. They are “alive” to the player, the main entities that she cares about. While not every one interacts with every other, they tend toward each one interacting with all the others.
Passive Interacting Objects: The Score and Scorekeeper are objects in the mix that do not interact with the active objects. Those two only interact with each other. They should neither see nor be seen by any other objects. Similarly for ShipMaker and WaveMaker in Asteroids, and PlayerMaker in Invaders. They observe the situation but they do not need to be seen by any other objects.; There are 13 implementors of interact_with_shipmaker and only two of them are not pass. That sort of thing is common. When I ripped through implementing all the abstract methods up above, I’d say that the average number of pass methods added to a class was around 9 or 10. That’s a lot of wasted space.; We could imagine that the hierarchy needs to be more refined, with groups of objects who interact with each other, or who interact very little, being off somewhere. But I think that leads quickly to a mess. The Ship, for example, interacts strongly with missiles and asteroids, but there are other object, like ShipMaker, that at least want to see if, and therefor must implement interact_with_ship. If we put that in the Active part of the hierarchy, don’t we get into some strange multiple inheritance solution or something? It seems messy.
Interaction Matrix: Readers have suggested that instead of these methods all being required, there should be some kind of centralized two-way interaction object that embodies all the interaction code. I object to that primarily on the grounds that it should be up to the Ship what the Ship does about things, not up to some godlike object that arrogates all the power to itself, trampling on the little guy in its relentless greedy quest for power. (Or was I thinking of [pick your favorite oligarch]?); But a matrix would at least be a central place to put consideration of these things, where we could focus our attention on it. There’s absolutely no question in my mind that this decentralized model is hard to think about. The universe is hard to think about too, so get over it. But it would be nice to make it easier to understand.
Single Message with Type: Suppose that, instead of just calling the selected double-dispatch method on everyone in the mix, we sent a single message to each object, something like

    some_flyer.interact("Missile", missile: Flyer, fleets)
    some_flyer.interact("Saucer", saucer: Flyer, fleets)

: Rather than encode the type in the message, interact_with_missile or interact_with_saucer, we’d encode it in a type parameter in a single message. Then the individual flyer could process the messages it cared about, with something like this:

class SomeFlyer(AsteroidFlyer):
    def interact(flyer_type: String, flyer: Flyer, fleets: Fleets):
        match flyer_type:
            case "Missile":
                fleets.remove(self)
            case _:
                pass

: I suppose we could even do that on the class itself::

class SomeFlyer(AsteroidFlyer):
    def interact(flyer, fleets):
        match flyer.__class__:
            case Missile:
                fleets.remove(self)
            case _:
                pass

: Now I was taught rather firmly that explicit dispatching on object class is a Never Do That, and a clear sign that we need specialized code in the subclasses. And whether we do it via __class__ or just a string, that’s clearly what we’re doing.; But it would be explicit, it would be compact, and it would extend automatically without need to edit subclasses, if we allowed them to use the _ default. It might even keep the interaction-specific code closer together and easier to inspect.
Publish-Subscribe: A more conventional approach might be a publish-subscribe sort of model. It’s not really fundamentally different from direct method dispatch, double dispatch, or the match/case above. They are all just ways of differentiating our handling of a bundle of similar events. It might look like this:

class SomeFlyer(AsteroidFlyer):
    def __init__(self):
        pubsub.subscribe("Missile Hit")

    def message(message_type, flyer, fleets):
        match message_type:
            case "Missile Hit":
                fleets.remove(self)
            case _:
                pass

Idea Summary (for Now)

We can see that they’re all pretty much the same, coming down to the details of the specific way we get into the object and what information we have at that point.

But the SingleMessage with Type and Publish-Subscribe approaches both provide for an explicit statement, in the object itself, of what it listens to and what it does when it hears something. They would reduce the span of all the classes, since they would no longer have to implement a dozen or more methods that they mostly don’t care about. These approaches would avoid deep-in-the-bag approaches, and they would be no less safe than what the developers (me) are doing in practice, pretty much blindly implementing default methods all over.

The individual classes would be smaller, and we would retain the notion of independent classes who don’t know much, if anything, about each other.

It might be worth trying. We would need to choose between these three very similar forms:

# Dispatch on Type String
class SomeFlyer(AsteroidFlyer):
    def interact(flyer_type: String, flyer: Flyer, fleets: Fleets):
        match flyer_type:
            case "Missile":
                fleets.remove(self)
            case _:
                pass

# Dispatch on actual Type
class SomeFlyer(AsteroidFlyer):
    def interact(flyer, fleets):
        match flyer.__class__:
            case Missile:
                fleets.remove(self)
            case _:
                pass

# Dispatch on Message Subject
class SomeFlyer(AsteroidFlyer):
    def __init__(self):
        pubsub.subscribe("Missile Hit")

    def message(message_type, flyer, fleets):
        match message_type:
            case "Missile Hit":
                fleets.remove(self)
            case _:
                pass

They’re rather similar, aren’t they? Maybe we’ll pick one and try it. We could probably figure out a way to slip one in, as we did with the metaclass, without changing the overall structure unless we like how things turn out.

Will we try that? I predict that we will. See you next time!