PubSub could hold onto objects that should be dead. Two attempts and much fumbling later, I’ve learned a bit but have no solution that I like.

In my general musing, I realized that the PubSub publish-subscribe mechanism will hold on to subscribers forever, even if they would otherwise be discarded. And they’d be receiving messages, which could cause issues. You can imagine a dead monster still delivering damage. Whatever might happen, it would likely not be what we intended.

You could argue that objects should be sure to unsubscribe before being lost, but a moment’s reflection on my fallibility tells me that that’s not a good bet. It turns out that there is a nice Python feature for this very situation, the WeakSet. As a warmup on this lovely Saturday, we’ll modify PubSub to use it. Here’s the class as it stands:

class PubSub:
    def __init__(self):
        self.subscriptions = dict()

    def subscribe(self, event, callback, one_shot=False):
        subscription = Subscription(callback, one_shot)
        try:
            self.subscriptions[event].append(subscription)
        except KeyError:
            self.subscriptions[event] = [subscription]

    def subscribe_once(self, event, callback):
        self.subscribe(event, callback, one_shot=True)

    def publish(self, event):
        try:
            subscriptions = list(self.subscriptions[event])
            for subscription in subscriptions:
                subscription.callback(event)
                if subscription.one_shot:
                    self.subscriptions[event].remove(subscription)
        except KeyError:
            pass

Currently, Subscriptions are kept in a list. We just need to change it to a WeakSet. I do this:

    def subscribe(self, event, callback, one_shot=False):
        subscription = Subscription(callback, one_shot)
        try:
            self.subscriptions[event].add(subscription)
        except KeyError:
            self.subscriptions[event] = WeakSet([subscription])

Sets take add I believe, rather than append. Tests break, what’s up with that?

Ah. This isn’t going to work. Subscriptions are created on the fly and only held in the subscriptions dictionary. So they quickly get collected. I’m going to have to go back to the drawing board on this idea. Reset –hard.

I’m a bit surprised to see the tests fail as they did: events were never called. It behaved as if a garbage collection had occurred between the subscribe and the publish in my tests, which were of course quite close together in time and space. A small mystery. In any case the WeakSet idea won’t hold water. Makes me glad that I had tests, that’s for sure.

Reflection

Let’s take a moment to think about what we might do about this. Not necessarily to try again, just to prime the thought pump. When we subscribe to an event, the event is currently and perhaps forever, just a string, the unique name of the event. We pass in a callback, which will have to be bound to the instance doing the subscribe, like this example:

class SecretKey:
    def placed(self, dungeon):
        self.visible = False
        def callback(event):
            dungeon.announce('A key has appeared!')
            self.visible = True
        dungeon.subscribe_once('button_pressed', callback)

The callback refers to self, which is the instance of SecretKey being placed. So long as SecretKey exists, that callback will address it. And as long as that callback is in the Subscription and the Subscription is in the PubSub subscriptions dictionary, SecretKey will exist even if we were to throw it away. In the case of SecretKey, we will be putting it in the player’s inventory, where it will remain, at least until she uses it up somehow. And we only subscribe_once, so PubSub will drop the subscription anyway.

If SecretKey was otherwise dropped, however, and was not unsubscribed, PubSub would keep it alive and keep sending it any messages it had subscribed to. Chaos could result.

What if … subscription had a weak reference to the callback, and what if there was a way to determine if it had been collected. I would guess that it would be set to None in that case, or something. Yes, according to a search it will return None.

We can try that quickly, I think.

class Subscription:
    def __init__(self, callback, one_shot: bool):
        self.callback = weakref.proxy(callback)
        self.one_shot = one_shot

The existing tests run. Let’s see if we can make one get collected.

Wow. After more fumbling than I’d have liked, I have a working test with a working weak reference.

    def called(self, event):
        self.fake_called = True

    def test_weakref_collected(self):
        pub_sub = PubSub()
        receiver = FakeSubscriber(self, pub_sub)
        self.fake_called = False
        pub_sub.publish('event')
        assert self.fake_called
        self.fake_called = False
        del receiver
        gc.collect()
        pub_sub.publish('event')
        assert not self.fake_called

Our FakeSubscriber sends called to the test instance, setting fake_called. We create it, it subscribes to ‘event’, we publish, it calls back. We delete it and collect garbage, publish event again and do not receive the callback. It’s working, but not good as we’ll see in a moment.

The code in PubSub gets slightly weird but it’s the way one deals with weak references, so we could get used to it:

class Subscription:
    def __init__(self, callback, one_shot: bool):
        self.callback = weakref.proxy(callback)
        self.one_shot = one_shot

class PubSub:
    def publish(self, event):
        try:
            subscriptions = list(self.subscriptions[event])
            for subscription in subscriptions:
                try:
                    subscription.callback(event)
                except ReferenceError:
                    pass
                if subscription.one_shot:
                    self.subscriptions[event].remove(subscription)
        except KeyError:
            pass

There’s no way to check a proxy to see if it is alive, so you can only try to use it and deal with the exception. That’s perhaps not too awful. But this troubles me more:

class FakeSubscriber:
    def __init__(self, send_to, pub_sub):
        self.send_to = send_to
        self.pub_sub = pub_sub
        def callback(event):
            self.send_to.called(event)
        self.hold = callback
        pub_sub.subscribe('event', callback)

See that self.hold? If that’s not there, the callback is not referenced by the FakeSubscriber instance, and it gets collected essentially instantly, and the test fails.

There’s just about no way we can remember always to hold onto our callback, so we can’t really live with this scheme. Reset –hard to get rid of this experiment.

I can think of more options:

  1. We could tell people to do it right and unsubscribe as needed. We’d have to implement unsubscribe, but that’s in the cards anyway.
  2. We could make all subscriptions one_shot, so that you’d have to resubscribe after each event, which might be more safe. Seems to me that we just get a new kind of common error, objects that should be responding but have stopped listening by accident.
  3. We could pass the subscribing object to PubSub and save it in the Subscription, with a weak proxy. Then we could either just check it and not send the callback, or we could require the callback to expect self back as its first parameter. That would, on the one hand, permit use of a method as a callback, not just an embedded def, which I mostly prefer, but on the other hand, the embedded def would then have to expect a self parameter.

Scheme #3 might not be too bad, and it would allow us to create callback methods as well as functions, which might be nice. With the present scheme, we cannot do that.

For now, two experiments, many missed tries, and 90 minutes later, the answer is, for now, #1, Do It Right Or Suffer The Consequences.

I’ve learned quite a bit about weak references, garbage collection, and—something I may forget before I need to know it again—learned that functions bound into a method do not persist, even if the object holding the method does persist. That surprises me, and I must confess that I’m not sure that’s what’s going on, although it is the only explanation I can come up with for why the self.hold trick worked.

Interesting morning. Time for my iced chai and perhaps a delicious granola bar. Kind of like a cereal and milk kit.