Python

How Function Decorators Work in Python

Understanding the way that function decorators are evaluated, executed and the importance of closures in executing them.

Nadim Jendoubi

Nadim Jendoubi

— 6 min read
How Function Decorators Work in Python
Decorators and Closures in Python

Function Decorators let us Tag or Mark functions in our code to alter their behavior in some way, this is the realm of Meta Programming and we are going to see how it works.

Function Decorators 101

We will start with an introduction to decorators and how are they evaluated in Python.

What is a Decorator?

A decorator is a function that allows a user to alter the functionality of an existing function by returning either an enhanced version of that function or simply replacing it by another.

Decorators are usually called before the definition of a function you want to decorate.

Basically this: 

@decorate
def A():
    print('running A()')

Is Equivalent to this: 

def A():
    print('running A()')
    
A = decorate(A)

In essence, decorators are just a way to simplify coding. Just as we observed, we can always call a decorator like any regular callable, by passing a function.

We can say that:
• A decorator is a function or another callable.
• A decorator may replace the decorated function with a different one.

When is a Function Decorator executed?

Function decorators are executed as soon as the module is imported, however, the decorated Functions are only executed when invoked at runtime.

This is what makes the difference. Since Decorators alter how functions behave, they have to be executed first i.e. when the module is imported in Python!

def decorate(func):
    print(f'running decorate({func})')
    return func
    
@decorate
def f1():
    print('running f1()')
    
def f2():
    print('running f2()')
    
    
def main():
    print('running main()')
    f1()
    f2()

if __name__ == '__main__':
    main()

# We execute the file with python3 testDecorator.py
# running decorate(<function f1 at 0x100631bf8>)
# running main()
# running f1()
# running f2()

We can notice that the decorate function ran as soon as we pressed "Enter" to execute! The module is imported, the decorator functions execute then the main function.

There are 2 points that  make our example above unusual:

  • A decorator is usually defined in one module and applied to functions in other modules for reusability, i.e. We don't define decorators where we define the functions to be decorated.
  • The decorator usually defines an inner function and returns it replacing the original passed function as its argument.

Most decorators replace the decorated function. They do this by relying on a very powerful feature called Closures to operate correctly.

Closures 101

Closures are widely used in Duck Typed Languages like Python and JavaScript and they are extremely useful, but to understand them we need to discuss Variable Scopes.

2 Rules of Variable Scopes

Python behaves following specific rules to decide on variable scopes during execution, we will review 2 rules as they are important to understand closures.

  • Python does not require us to declare variables, however, it assumes that any variable assigned in the body of a function is local, so this example will raise an exception:
b = 6
def f2(a):
    print(a)
    print(b)
    b = 9
    
f2(3)
# UnboundLocalError: local variable 'b' referenced before assignment
  • Treating a Variable as a Global Variable is done by adding the global keyword. This is due to having multiple scopes: the Global Scope and the Function Local Scope.
b = 6
def f2(a):
    global b
    print(a)
    print(b)
    b = 9
    
f2(3)
# 3
# 6
print(b)
# 9

What is a Closure?

A Closure is a function with an extended scope that encompasses variables referenced in its body coming from the local scope of an outer function that encompasses it.

Basically, the closure can access variables that are not local or global but are local to the outer surrounding functions.

Here's an example of a function that calculates the average of a growing series of values from the book Fluent Python:

def make_averager():
    series = []
    
    def averager(new_value):
        series.append(new_value)
        total = sum(series)
        return total / len(series)
        
    return averager

When invoked, make_averager returns an averager function object. Each time an
averager is called, it appends the passed argument to the series, and computes the
current average.

We can notice that series is a local variable of make_averager, and within averager, series is a free variable, it is not bound in the local scope of averager.

The closure for averager function
The closure for averager function. Image from the book Fluent Python

Free Variables and the nonlocal Keyword

Free variables are variables that are not bound to local scope and should always be mutable so they can be modified by the inner function, they are kept in the __closure__ attribute of the returned function and declared with the nonlocal keyword.

We can change our implementation of make_averager like this:

def make_averager():
    count = 0
    total = 0
    
    def averager(new_value):
        nonlocal count, total
        count += 1
        total += new_value
        return total / count
        
    return averager

If we tried executing this without the "nonlocal count, total" line we would've had an exception as Integers are immutable in nature.

How does Python find Variables?

The Python Compiler defines how a variable x is fetched when a function is defined based on:

  • If a global x declaration exists, x is taken from and assigned to the x global variable module.
  • If a nonlocal x declaration exists, x is taken from and assigned to the x local
    variable of the nearest surrounding function where x is defined.
  • If x is a parameter or is assigned a value in the function body, then x is the local
    variable.

Implementing a Function Decorator

The Decorator Pattern in the book Design Patterns by Erich Gamma is explained in detail. Although it is different from decorators in Python, we still can quote this about decorators:

Attach additional responsibilities to an object dynamically.

Let's Implement an example of a clocking decorator that computes how much time it takes to execute the decorated function to see how much that quote applies! 

import time
import functools

def clock(func):

     @functools.wraps(func)
     def clocked(*args, **kwargs):
         t0 = time.perf_counter()
         result = func(*args, **kwargs)
         elapsed = time.perf_counter() - t0
         name = func.__name__
         arg_lst = [repr(arg) for arg in args]
         arg_lst.extend(f'{k}={v!r}' for k, v in kwargs.items())
         arg_str = ', '.join(arg_lst)
         print(f'[{elapsed:0.8f}s] {name}({arg_str}) -> {result!r}')
         return result
     
     return clocked

The decorator @functools.wraps is one of the ready-to-use decorators in the standard library, this particular one allows us to pass the keyword arguments (**kwargs) and the __name__ and __doc__ of the decorated function. In this instance we can copy the relevant attributes from func to clocked.

import time
from clockdecorator import clock

@clock
def snooze(seconds):
    time.sleep(seconds)

@clock   
def factorial(n):
    return 1 if n < 2 else n*factorial(n-1)
    
if __name__ == '__main__':
    print('Calling snooze(.123)')
    snooze(.123)
    print('Calling factorial(6)')
    print('6! =', factorial(6))
    
# Executing the file test.py 

# Calling snooze(.123)
# [0.12363791s] snooze(0.123) -> None
# Calling factorial(6)
# [0.00000095s] factorial(1) -> 1
# [0.00002408s] factorial(2) -> 2
# [0.00003934s] factorial(3) -> 6
# [0.00005221s] factorial(4) -> 24
# [0.00006390s] factorial(5) -> 120
# [0.00008297s] factorial(6) -> 720
# 6! = 720

This is how a decorator normally functions: it replaces the decorated function with an alternate one that takes the same arguments and usually produces the same results, while also doing some added processing.

Parametrized Decorators

We saw how to implement a decorator, however sometimes we need to pass arguments to the decorator function, so how can we do it?

The answer is to Implement a Decorator Factory! Basically a Decorator that returns decorators.

Here's the clock decorator parametrized:

import time

DEFAULT_FMT = '[{elapsed:0.8f}s] {name}({args}) -> {result}'

def clock(fmt=DEFAULT_FMT):
    def decorate(func):
        def clocked(*_args):
            t0 = time.perf_counter()
            _result = func(*_args)
            elapsed = time.perf_counter() - t0
            name = func.__name__
            args = ', '.join(repr(arg) for arg in _args)
            result = repr(_result)
            print(fmt.format(**locals()))
            return _result
        return clocked
    return decorate
    
@clock('{name}({args}) dt={elapsed:0.3f}s')
def snooze(seconds):
    time.sleep(seconds)

Here the clock is The Decorator Factory, decorate is the actual decorator and the clocked function is the wrapper for the inner function of the decorator. Closures allow us access to the parameter fmt and we can use it in the inner function.

Conclusion

Today we covered an advanced topic but in reality it cannot be covered by one article post! I tried to summaries what the original author of the book Fluent Python had to say about the subject, however I do advise you to research it more and here are some references that could help you.

Further reading

Read more