P-264 - Decorator Tutorial

Let me explain. No, it is too much. Let me sum up.

Some Fundamentals

We’ll briefly discuss the fundamental idea of Python decorators, and the related supporting notions of functions as objects and function wrappers.

Decorator Notion

The Python language includes the notion of a “decorator”, typically represented as a line, beginning with an at sign, prefixing a function or class definition, perhaps like this:

    def test_property(self):
        class Mumble:
            def __init__(self, foo):
                self._secret_foo = foo

            @property
            def foo(self):
                return self._secret_foo

        mumble = Mumble(42)
        assert mumble.foo == 42
        with pytest.raises(AttributeError):
            mumble.foo = 37

The example above allows the programmer to refer to a property foo in instances of Mumble class by saying

    my_mumble.foo

But as we see in the test, the programmer cannot store into the (illusory) property foo: Python will not allow it. In addition, the PyCharm IDE is smart enough to flag that statement in the editor, before you even try to run it:

python diagnostic

We’ll not debate whether and when to use that particular decorator. We are here to describe what decorators are, and how to build our own.

Functions Are Objects

Everything in Python is an object, and a function is no exception. Functions, once created, can be passed as arguments and, in general, handled just like any other object, subject to whatever constraints the individual object has. A function can be saved in a variable and called later. Here is a meaningless but real example.

def hello():
    print("hello")

def goodbye():
    print("goodbye")

greeting = hello
greeting()
greeting = goodbye
greeting()

The code above is a long way of causing Python to print:

hello
goodbye

The variable greeting appears to be, and is, just like any other variable. It could contain an integer, a Person object, an Email object: anything. In our example, it contains functions, and therefore it can be called by saying greeting().

Initially greeting contains (a pointer to) the function hello, so when we first say greeting(), Python calls hello and prints “hello”. Then we set greeting to point to goodbye, so the next call to greeting prints “goodbye”.

Function Wrappers

A wrapper is an object that has a reference to another function and can call it at will. Here is a simple example:

    def test_wrapper(self):
        def walking():
            return "walking"

        def running():
            return "running"

        def mumbling():
            return "mumbling"

        def wrapper(func):
            return "you are " + func() + ", sir!"

        assert wrapper(walking) == "you are walking, sir!"
        assert wrapper(running) == "you are running, sir!"
        assert wrapper(mumbling) == "you are mumbling, sir!"

Our simple wrapper is a function (remember, a function is an object) receives a function as its parameter, and calls the function in the middle of the wrapper’s own functioning.

Note that the wrapper is called with just the name of the function, e.g. walking, not the result of a call to the function, e.g. walking(). (Yes, if the output were all that we wanted, we could have written a simpler program. It’s an example. Settle down.)

Decorators - What are they?

A Python decorator is a construct that associates additional behavior with the functions to which it is applied.

Decorators can be written to do anything we can think of, and some of the ones available for Python do truly amazing and complicated things. Like any other function or object, the sky’s the limit on what we can make them do.

Here’s an imaginary example of a decorator without parameters, and one with parameters.

@do_twice
def hello(name):
    print("hello, " + name)

@do_n_times(3)
def goodbye(name):
    print("goodbye, " + name)

hello("Alex")
goodbye("Alex")

With suitable definitions of do_twice and do_n_times, the output of this program would be:

hello, Alex
hello, Alex
goodbye, Alex
goodbye, Alex
goodbye, Alex

Every use of the decorated function hello would result in two calls to hello, because it was decorated by @do_twice, and because @do_twice was written so as to execute whatever function it held, twice. Similarly, because we gave do_n_times a parameter of 3, that one is done three times.

The creation of a decorator always follows one of a few standard forms. There are two basic approaches: We can define a decorator using a class, or we can define it using a function. The two forms break down further, into the case of a decorator that itself takes no parameters, and a decorator that does take its own parameters.

In what follows we will show, using tests, class-based decorators, simple and parameterized, and then function-based decorators, also simple and parameterized:

Simple decorator implemented via a class;
Parameterized decorator implemented via a class;
Simple decorator implemented via a function;
Parameterized decorator implemented via a function;

Decorators Implemented via Class

Historically, it seems that Python decorators are introduced via nested wrapping functions. We’ll break the pattern here. In my view, the class-oriented implementation is simpler and more clear. YMMV, as always, and as always, you should choose the approach that you see as better for you.

Simple Decorator via Class

Précis: Implement a class with the desired decorator name. In __init__ receive and save the function to be wrapped. In __call__, receive *args* and **kwargs, and implement the desired decorator behavior and return value.; The methods __init__ and __call__ are required and must have those names.

    def test_do_twice_class(self):
        class do_twice:
            def __init__(self, func):
                self.func = func

            def __call__(self, *args, **kwargs):
                result = "do_twice begin\n"
                result += self.func(*args, **kwargs)
                result += self.func(*args, **kwargs)
                result += "end do_twice\n"
                return result

        @do_twice
        def hello(name):
            return "hello, " + name + "!\n"

        said_twice = hello("Avery")
        assert (said_twice ==
"""do_twice begin
hello, Avery!
hello, Avery!
end do_twice
""")

Recipe / Description

1: Define Decorator Class: The decorator class is given the exact name of the desired decorator.
2: Define method __init__: The decorator class’s __init__ method receives and saves the specific function to be wrapped.
3: Define method __call__: The decorator class’s __call__ method will be invoked on each call of any decorated function. It receives the general parameters *args and **kwargs. This allows the decorator to accept any function no matter what its parameters are.
4: Implement Decorator Behavior: In the __call__ function, implement whatever behavior the decorator is supposed to do. Call the decorated function as appropriate.
5: Return Result: Typically, the decorator carries out its own activities and then returns the actual result of calling the decorated function. In our tests, we create and return a string result for testing purposes. This is not typical of most decorators.
In Sum: For a simple decorator without parameters, implemented as a class, the recipe is quite simple. We merely have to implement __init__ to receive the function to be wrapped, and __call__ to receive the actual arguments to the function on each call, implement the behavior, and call the function as appropriate.

Parameterized Decorator Using Class

Précis: Implement a class with the desired decorator name. In __init__, receive and saves the decorator parameters. In __call__, receive and save the wrapped function, returning the method perform_behavior. Implement perform_behavior to receive *args* and **kwargs, implementing the desired behavior, calling the decorated function as appropriate, returning desired results.; The methods __init__ and __call__ are required and must have those names. The method perform_behavior can have any name you choose. I chose that one.

    def test_do_n_times_class(self):
        class do_n_times:
            def __init__(self, number_of_times):
                self.number_of_times = number_of_times

            def __call__(self, func):
                self.func = func
                return self.perform_behavior

            def perform_behavior(self, *args, **kwargs):
                result = "do_n_times begin\n"
                for _ in range(self.number_of_times):
                    result += self.func(*args, **kwargs)
                result += "end do_n_times\n"
                return result

        @do_n_times(3)
        def hello(name):
            return "hello, " + name + "!\n"

        said_twice = hello("Parker")
        assert (said_twice ==
"""do_n_times begin
hello, Parker!
hello, Parker!
hello, Parker!
end do_n_times
""")

What follows relates to the simple case in a quite parallel fashion to the relationship between simple and parameterized decorators defined via functions: We simply add one more level of call.

Recipe / Description

1: Define Decorator Class: The decorator class is given the exact name of the desired decorator.
2: Define method init: The decorator class’s __init__ method receives and saves the decorator’s own actual parameters
3: Define method __call__: The decorator class’s __call__ method will be invoked once for each decorated function. It receives and saves the specific function to be decorated. The call function returns a reference to the class’s perform_behavior method:
4: Define method perform_behavior: The perform_behavior method will be invoked on each call of any decorated function. It receives arbitrary parameters *args* and **kwargs, allowing us to wrap any function no matter what its parameters are/
5: Implement Decorator Behavior: In the perform_behavior method,implement whatever behavior the decorator is supposed to do. Call the decorated function as appropriate.
5: Return Result: Typically, the decorator carries out its own activities and then returns the actual result of calling the decorated function. In our tests, we create and return a string result for testing purposes. This is not typical of most decorators.
In Sum: For a decorator with parameters implemented as a class, the recipe is similar to that for a simple decorator, except that we introduce one additional method, which we call here perform_behavior, in addition to the __init__ and __call__ as before. The __init__ receives and saves the decorator’s parameters, while the __call__ receives and saves the function to be decorated. The perform_behavior method receives the function’s actual parameters.and carries out the decorator behavior, calls the decorated function as needed, and returns desired results.

Simple vs Parameterized Decorators

In the class-based approach, the difference between simple and parameterized decorators comes down to implementing methods to receive, in order, the decorator parameters, the function to be decorated, and the actual parameters of a function call. __init__ and __call__ are used for the first two, and a method whose name the programmer can pick handles the third.

Decorators Implemented via Function

This seems to be the most common approach we find written up, which makes me believe it’s perhaps the first way Python offered, Be that as it may, I find the class-oriented approach described above to be far simpler and more clear. I offer these examples to show how decorators can be defined with functions, without classes, and of course developers are free to do things as they see fit.

Simple Decorator via Function

Précis: The decorator function has the same name as the decorator. It receives the function to be decorated as its sole parameter. The decorator function defines and returns a wrapper function that receives the actual arguments of a specific use of the decorated function. The wrapper performs the desired decorator actions, calling the wrapped function as appropriate.

    def test_do_twice(self):
        def do_twice(func): 
            @functools.wraps(func)
            def wrapper(*args, **kwargs):
                result = "do_twice begin\n"
                result += func(*args, **kwargs)
                result += func(*args, **kwargs)
                result += "end do_twice\n"
                return result
            return wrapper

        @do_twice
        def hello(name):
            return "hello, " + name + "!\n"

        said_twice = hello("Avery")
        assert (said_twice ==
"""do_twice begin
hello, Avery!
hello, Avery!
end do_twice
""")

Recipe / Description

1: Define the Decorator Function: We create a function with the exact name we want our decorator to have, do_twice, accepting one parameter, func, which will be the function to be wrapped. This function is invoked once for each function decorated.
2: Define and Return the Wrapper Function: The decorator function’s job is to define and return another function, here called wrapper, which does the work of the decorator. This wrapper function is called for each invocation of the decorated function.
3: Wrapper Accepts *args and **kwargs: We generally build our decorator wrapper to receive arbitrary arguments *args and **kwargs. This will allow any function to be wrapped no matter what its arguments may be. It is possible to specify the arguments if needed, but that’s not the usual case.
4: Implement Decorator Behavior: In the wrapper function, implement whatever behavior the decorator is supposed to do. Here, we just append together the result of calling the provided function func, twice, enclosed in a couple of strings of our own.
5: Return Result: Return the result that we intend to have the wrapped function provide. In our case we chose to return the appended results plus some words of our own. In a logging decorator, we might write to a log and just return the function’s normal result. Probably most decorators only call the function once, and just return the function’s result.
6: Use functools.wraps: The polite, pythonic practice is to prefix our wrapper with the functools.wraps(func) line as shown. The only effect of this is to change the name of our wrapper function to the name of the main function, so that various prints and debug output mention the real function rather than wrapper. Everything works the same without this line, so long as things go right. When things go wrong, the output is nicer with the line in.
In Sum: For a simple decorator with no parameters, implemented as a function, the above is the standard recipe. We define the decorator name as a function accepting a function parameter; we define a wrapper function accepting *args and **kwargs; we implement the decorator behavior, returning the desired function result. We decorate our wrapper with wraps, and our decorator function returns the wrapper as its result.

Parameterized Decorator via Function

Précis: The decorator function has the same name as the decorator. It receives the parameters of the decorator. The decorator defines and returns a function with a single parameter, the specific function to be wrapped. The function-receiving function defines and returns a wrapper function that receives the actual arguments of a specific use of the decorated function. The wrapper performs the desired decorator actions, calling the wrapped function as appropriate.

    def test_repeat_parameterized(self):
        def do_n_times(number_of_times):
            def get_function(func):
                @functools.wraps(func)
                def wrapper(*args, **kwargs):
                    result = "do_n_times begin\n"
                    for _ in range(number_of_times):
                        result += func(*args, **kwargs) #
                    result += "end do_n_times\n"
                    return result
                return wrapper
            return get_function

        @do_n_times(3)
        def hello(name):
            return "hello, " + name + "!\n"

        said_twice = hello("Parker")
        assert (said_twice ==
"""do_n_times begin
hello, Parker!
hello, Parker!
hello, Parker!
end do_n_times
""")

Recipe / Description

We’ll go through this bit by bit, but note that it is really just the same as the simple case, except that there is an additional level of nested functions, so as to pick up the decorator’s own parameters before we get going.

1: Define the Decorator Function: We create a function with the name we want the decorator to have, do_n_times, receiving the decorator’s own concrete parameters. This function is called once for each function decorated.
2: Define and Return a Function to Get The Decorated Function: This first-level inner function receives the actual function to be wrapped. This function is called once for each function decorated.
3: Define and Return the Wrapper Function: The function receiver’s job is to define and return a second-level function, here called wrapper, which does the work of the decorator. This function is called each time the decorated function is invoked.
4: Wrapper Accepts *args and **kwargs: We generally build our decorator wrapper to receive arbitrary arguments *args and **kwargs. This will allow any function to be wrapped no matter what its arguments may be. It is possible to specify the arguments if needed, but that’s not the usual case.
5: Implement Decorator Behavior: In the wrapper function, implement whatever behavior the decorator is supposed to do. Here, we just append together the result of calling the provided function func, twice, enclosed in a couple of strings of our own.
6: Return Result: Return the result that we intend to have the wrapped function provide. In our case we chose to return the appended results plus some words of our own. In a logging decorator, we might write to a log and just return the function’s normal result. Probably most decorators only call the function once, and just return the function’s result.
7: Use functools.wraps: The polite, pythonic practice is to prefix our wrapper with the functools.wraps(func) line as shown. The only effect of this is to change the name of our wrapper function to the name of the main function, so that various prints and debug output mention the real function rather than wrapper. Everything works the same without this line, so long as things go right. When things go wrong, the output is nicer with the line in.
In Sum: For a simple decorator with parameters, implemented as a function, the above is the standard recipe. We define the decorator name as a function accepting the decorator’s actual parameters; that function defines and returns an inner receiving a function parameter; that function defines the wrapper function accepting *args and **kwargs; we implement the decorator behavior, returning the desired function result. We decorate our wrapper with wraps, and our decorator’s get_function returns the wrapper as its result.

Simple vs Parameterized Decorators

As you can hopefully see, the case of a decorator with parameters merely adds one level of function nesting to the simpler case without decorator parameters. Personally, I find that additional level disproportionately hard to think about, but the test code above is easy to modify for any specific decorator that I might need. I provide the detailed steps more as a description / explanation of the process, though they can be followed as a recipe.

Decorating Classes

We can put a decorator above a class definition if we choose. What does that mean?

A decorator applied to a class definition simply wraps the class’s __init__ method, no more and no less. Therefore such a decorator needs to return the result of the init, which implies that the decorator will at some point create an instance of the decorated class. We might return it, or we might return some other class containing it.

We will not explore this further here. The essential notion is that we’re wrapping the __init__, and all else follows from that.

In Actual Play

I would emphasize that I find the class form of decorator far more clear and easier to produce than the function form. If I gave advice, which frequent readers know I do not, I would advise using the class form.

In either case, in actual use, the decorator class or function will typically be defined at the top level, along with other classes and top-level functions.

Our decorators here returned special testing-oriented results. In actual use, decorators are usually invisible to the functions wrapped, returning exactly the same results that the original function returns, implementing the decorator behavior “off to the side”, perhaps logging results or building specialized structures. There is no practical limit to what decorators can do: it’s up to the imagination and creativity of the programmer.

I would caution that because their behavior is hidden, and because they are a bit deeper in the bag of tricks, decorators should be used with caution. Naturally, the wise developer will experiment with them a lot before committing to their use in production code.

Feedback

I would particularly welcome feedback on this article, via email to ronjeffries at acm dot org or on mastodon dot social, @ronjeffries. I will try to update this article based on feedback to improve it.

Meta

Some Fundamentals

Decorator Notion

Functions Are Objects

Function Wrappers

Decorators - What are they?

Decorators Implemented via Class

Simple Decorator via Class

Parameterized Decorator Using Class

Simple vs Parameterized Decorators

Decorators Implemented via Function

Simple Decorator via Function

Parameterized Decorator via Function

Simple vs Parameterized Decorators

Decorating Classes

In Actual Play

Feedback