Python

Generic Types, Covariance and Overloading Signatures in Python

In depth Review of Covariance, Contravariance and Invariance with Generic Classes in Python.

Nadim Jendoubi

May 5, 2023 — 6 min read

Generic Types, Covariance and Overloading Signatures in Python

We have visited the topics of protocols and classes in this series before, but we haven't really talked about generic classes, variance and overloading in depth. It is time we did as these are not so easy to wrap one's head around!

We will start with figuring out how to overload method signatures, then we'll move to Generic classes and Variance.

How to Explicitly Overload Method and Function Signatures?

What makes Python powerful is the fact that we can pass any argument of any type any time we want without many constraints, but sometimes this can be our own undoing as certain use cases require us to implement hard typed code, i.e. we need our methods to be annotated so we can control what will it accept as arguments, basically we control the types of the passed arguments.

The overload decorator provided by the typing module solves this specific dilemma:

from typing import overload, Union, TypeVar
import functools
import operator
from collections.abc import Iterable

T = TypeVar('T')
S = TypeVar('S')

@overload
def sum(it: Iterable[T]) -> Union[T, int]: ...

@overload
def sum(it: Iterable[T], /, start: S) -> Union[T, S]: ...

def sum(it, /, start=0):
    return functools.reduce(operator.add, it, start)

The first overloaded signature is for the simplest use case, we take an iterable of type T and return a result of type T or int.
The second overloaded signature is for a different use case, we pass an iterable of type T and an initializer value of type S in case the iterable is empty it serves as the default result.
The last function is the actual function implementation we call, as you see it takes a lot of signature overloads to annotate a simple function.

There's a reason why Python is duck typed, it's simpler and more flexible that way without dealing with cumbersome type hinting.

Pros and Cons of Overloading signatures

Overloading signatures allows static type checkers to flag our code in case we make unsupported type calls to our function which can save us time when debugging.
A key selling point of overloading is declaring the return types as precisely as possible, can help us avoid the isinstance checks or any possible typing errors.
Overloading signatures means more code, more lines to be maintained by developers.
Code is more restrictive and way less flexible due to the enforced type hinting.

How to Implement a Generic Class?

In our example to implement a Generic Class, we will use the previously defined Tombola class when we discussed protocols and we will create a generic class that subclasses it: LottoBlower.

import abc
import random
from collections.abc import Iterable
from typing import TypeVar, Generic

class Tombola(abc.ABC):

    @abc.abstractmethod
    def load(self, iterable):
        """Add items from an iterable."""
    
    @abc.abstractmethod
    def remove(self):
        """Remove item at random, returning it."""
    
    def loaded(self):
        return bool(self.inspect())

    def inspect(self):
        items = []
        while True:
            try:
                items.append(self.pick())
            except LookupError:
                break
        self.load(items)
        return tuple(items)
        


T = TypeVar('T')
class LottoBlower(Tombola, Generic[T]):
    def __init__(self, items: Iterable[T]) -> None:
        self._balls = list[T](items)
    
    def load(self, items: Iterable[T]) -> None:
        self._balls.extend(items)
        
    def pick(self) -> T:
        try:
            position = random.randrange(len(self._balls))
        except ValueError:
            raise LookupError('pick from empty LottoBlower')
        return self._balls.pop(position)

    def loaded(self) -> bool:
        return bool(self._balls)

    def inspect(self) -> tuple[T, ...]:
        return tuple(self._balls)

Generic classes tend to use multiple inheritances from multiple interfaces and generic classes, that's why the example portrays this. We return T typed results and the arguments that are passed are also T typed.

Invariance, Covariance and Contravariance in Python Generic Classes

In practice, variance is complex concept that's mostly used by library authors who want to support new generic container types or provide callback-based APIs.

In a brief explanation, here's what to retain:

If B is a subtype of A, then a generic type constructor GenType is called:

Covariant, if GenType[B] is compatible with GenType[A] for all classes A and B.
Contravariant, if GenType[A] is compatible with GenType[B] for all classes A and B.
Invariant, if neither of the above is true.

Invariance in Python

Imagine that a school cafeteria has a rule that only juice dispensers can be installed. General beverage dispensers are not allowed because they may serve sodas, which are banned by the school board.

from typing import TypeVar, Generic

class Beverage:
    """Any beverage."""

class Juice(Beverage):
    """Any fruit juice."""

class OrangeJuice(Juice):
    """Delicious juice from Brazilian oranges."""

T = TypeVar('T')

class BeverageDispenser(Generic[T]):
    """A dispenser parameterized on the beverage type."""

    def __init__(self, beverage: T) -> None:
        self.beverage = beverage

    def dispense(self) -> T:
        return self.beverage

def install(dispenser: BeverageDispenser[Juice]) -> None:
    """Install a fruit juice dispenser."""
    print("yep I can do this")

juice_dispenser = BeverageDispenser(Juice())
install(juice_dispenser)
# yep I can do this

beverage_dispenser = BeverageDispenser(Beverage())
install(beverage_dispenser)
## mypy: Argument 1 to "install" has
## incompatible type "BeverageDispenser[Beverage]"
## expected "BeverageDispenser[Juice]"

orange_juice_dispenser = BeverageDispenser(OrangeJuice())
install(orange_juice_dispenser)
## mypy: Argument 1 to "install" has
## incompatible type "BeverageDispenser[OrangeJuice]"
## expected "BeverageDispenser[Juice]"

A generic type L is invariant when there is no supertype or subtype relationship
between two parameterized types A and B. Basically if L is invariant, then L[A] is not a supertype or a subtype of L[B]. They are inconsistent in both ways

In this case, the Generic Constructor BeverageDispenser(Generic[T]) is invariant as it's not compatible with BeverageDispenser[Beverage] and not compatible with BeverageDispenser[OrangeJuice].

Covariance in Python

To make the dispenser class able to accept any type of beverage and its subtypes, we have to make it covariant for greater class flexibility.

from typing import TypeVar, Generic

class Beverage:
    """Any beverage."""

class Juice(Beverage):
    """Any fruit juice."""

class OrangeJuice(Juice):
    """Delicious juice from Brazilian oranges."""

T_co = TypeVar('T_co', covariant=True)

class BeverageDispenser(Generic[T_co]):
    """A dispenser parameterized on the beverage type."""

    def __init__(self, beverage: T_co) -> None:
        self.beverage = beverage

    def dispense(self) -> T_co:
        return self.beverage

def install(dispenser: BeverageDispenser[Juice]) -> None:
    """Install a fruit juice dispenser."""
    print("yep I can do this")

juice_dispenser = BeverageDispenser(Juice())
install(juice_dispenser)
# yep I can do this

orange_juice_dispenser = BeverageDispenser(OrangeJuice())
install(orange_juice_dispenser)
# yep I can do this

beverage_dispenser = BeverageDispenser(Beverage())
install(beverage_dispenser)
## mypy: Argument 1 to "install" has
## incompatible type "BeverageDispenser[Beverage]"
## expected "BeverageDispenser[Juice]"

For covariant types, if we have 2 classes A and B, and B is a subclass of A, then a generic type C is covariant when C[A] is a supertype to C[B].

Covariance follows the subtype relationship of the actual type parameters, i.e. if you can invoke it on the parent you can invoke it on its children but you can't invoke it on the parent's parent.

Contravariance in Python

The contravariance is similar to the covariance relationship only it works backwards, i.e. Contravariant generic types reverse the subtype relationship of the actual type parameters.

from typing import TypeVar, Generic

class Beverage:
    """Any beverage."""

class Juice(Beverage):
    """Any fruit juice."""

class OrangeJuice(Juice):
    """Delicious juice from Brazilian oranges."""

T_contra = TypeVar('T_contra', contravariant=True)

class BeverageDispenser(Generic[T_contra]):
    """A dispenser parameterized on the beverage type."""

    def __init__(self, beverage: T_contra) -> None:
        self.beverage = beverage


def install(dispenser: BeverageDispenser[Juice]) -> None:
    """Install a fruit juice dispenser."""
    print("yep I can do this")

juice_dispenser = BeverageDispenser(Juice())
install(juice_dispenser)
# yep I can do this

beverage_dispenser = BeverageDispenser(Beverage())
install(beverage_dispenser)
# yep I can do this

orange_juice_dispenser = BeverageDispenser(OrangeJuice())
install(orange_juice_dispenser)
## mypy: Argument 1 to "install" has
## incompatible type "BeverageDispenser[OrangeJuice]"
## expected "BeverageDispenser[Juice]"

If we have 2 classes A and B, and B is a subclass of A, then a generic type C is Contravariant when C[A] is a subtype to C[B].

The author of the book Fluent Python gave us some rule of thumbs to follow when we're dealing with Variance, I will quote them here:

If a formal type parameter defines a type for data that comes out of the object, it can be covariant.
If a formal type parameter defines a type for data that goes into the object after its initial construction, it can be contravariant.
If a formal type parameter defines a type for data that comes out of the object and the same parameter defines a type for data that goes into the object, it must be invariant.
To err on the safe side, make formal type parameters invariant.

Conclusion

Generic classes can solve multiple problems and add a level of abstraction to our code at the cost of the extra maintenance and care we need to develop an API with zero bugs that can be too subtle to notice.