Python

What Makes a Python Object? Part II

How to Make our Objects Behave as Built-In Python Objects by Implementing Python Magic Methods

Nadim Jendoubi

Nadim Jendoubi

— 5 min read
What Makes a Python Object? Part II
What Makes a Python Object? 

In the previous part of the 2 part series of this article, we answered the question of how can it be possible for user-defined types to behave as naturally as the built-in types  through the Data Model.

We discussed the possible Special Method Implementations to represent our objects, with the help of __repr__ for debugging and __str__ for printing to end users. We also explained the __bytes__ to obtain the byte representation and finally the __format__ used by f-strings, by the built-in function format(), and by the str.format() method.

We also added an alternative constructor by implementing a class method with the @classmethod decorator and finally we made it possible to use our vector2d objects in sets and dictionary keys by making them hashable.

Today we will build on that and answer two more questions:

  • How to make our attributes private or read only?
  • How to use __slots__ to save memory?

We will make use of the same class we implemented before, the Vector2d class.

Use case: Vector2d Class so far

from array import array
import math

class Vector2d:
    typecode = 'd'
    __match_args__ = ('x', 'y')
    def __init__(self, x, y):
        self.__x = float(x)
        self.__y = float(y)

    def __iter__(self):
        return (i for i in (self.x, self.y))

    def __repr__(self):
        class_name = type(self).__name__
        return '{}({!r}, {!r})'.format(class_name, *self)

    def __str__(self):
        return str(tuple(self))

    def __bytes__(self):
        return (bytes([ord(self.typecode)]) + bytes(array(self.typecode, self)))
    
    def __eq__(self, other):
        return tuple(self) == tuple(other)

    def __abs__(self):
        return math.hypot(self.x, self.y)

    def __bool__(self):
        return bool(abs(self))

    @classmethod
    def frombytes(cls, octets):
        typecode = chr(octets[0])
        memv = memoryview(octets[1:]).cast(typecode)
        return cls(*memv)

    def angle(self):
        return math.atan2(self.y, self.x)

    def __format__(self, fmt_spec=''):
        if fmt_spec.endswith('p'):
            fmt_spec = fmt_spec[:-1]
            coords = (abs(self), self.angle())
            outer_fmt = '<{}, {}>'
        else:
            coords = self
            outer_fmt = '({}, {})'
        components = (format(c, fmt_spec) for c in coords)
        return outer_fmt.format(*components)

    @property
    def x(self):
        return self.__x

    @property
    def y(self):
        return self.__y
        
    def __hash__(self):
        return hash((self.x, self.y))

How to make our attributes private or read only?

The short answer is we can't. Python has no equivalent to Java's private modifier to create private variables. However, Python offers a straightforward way to block unintended changes to “private” and "protected" attributes in a class or a subclass.

This mechanism is called Name Mangling, In Wikipedia, it is defined as:

In Python, mangling is used for class attributes that one does not want subclasses to use which are designated as such by giving them a name with two or more leading underscores and no more than one trailing underscore.

Let's explain with an example, imagine that we wrote a class named Person that  has an age attribute that we want to remain private or protected. We subclass Person to create Employee, if by any chance we create an age attribute in Employee we will have a clash between the 2 attributes, one of them will clobber the other.

Python has a way around this, we could create our age attribute with 2 leading underscores such as __age and this way Python will store the variable in the instance __dict__, so our __age becomes _Person__age in the Person class and it becomes _Employee__age in the Employee class. Behold the Power of Name Mangling.

Name Mangling is a way to secure our attributes from accidental mutation, not malicious prying, anybody who is aware of how private names are distorted can peruse the attributes without obstruction.

Other than the automatic name mangling with the double underscores, there's a different convention among Python programmers, using a  single underscore prefix that has no significance to the Python interpreter but when used in an attribute name it marks the attribute as private or protected and should not be accessed from outside the class.

The practice of “protecting” attributes by convention with the form self._x is widespread but bear in mind that the attributes are not actually private or immutable.

Now we already have our __x and __y attributes in the Vector2d Class, we can test the access to our  secure attributes like this: 

>>> v1 = Vector2d(3, 4)
>>> v1.__dict__
{'_Vector2d__y': 4.0, '_Vector2d__x': 3.0}
>>> v1._Vector2d__x
3.0

How to use __slots__ to save memory?

When we create an instance of an object, by default, Python stores the attributes of the instance in a dictionary named __dict__. Dictionaries have a significant memory overhead, i.e. dictionaries use more memory, whether we try to optimize or not  we still can feel it.

For this reason, Python offered us the __slots__, which is an attribute we define to hold our attribute names instead of the usual __dict__ and the attributes named in __slots__ are stored in a hidden array or references that use less memory than a dictionary.

Let's see the differences between the two approaches. Here's how we normally define a class: 

# Test with normal dictionary holding our attributes
class TestWithDict(object):
      def __init__(self, *args, **kwargs):
                self.a = 1
                self.b = 2
  
if __name__ == "__main__":
     instance = TestWithDict()
     print(instance.__dict__)
# {'a': 1, 'b': 2}

Now here's the same class with the __slots__ attribute: 

# Test with __slots__ holding our name attributes
class TestWithSlots(object):
      __slots__=['a', 'b']
      def __init__(self, *args, **kwargs):
                self.a = 1
                self.b = 2
  
if __name__ == "__main__":
     instance = TestWithSlots()
     print(instance.__slots__)
# ['a', 'b']

There are few caveats that we need to go through:

  • If we declare the __slots__ attribute, the instance will accept values for those specific attribute named in __slots__ only.
  • The __slots__ attribute must be present when the class is created as adding or changing it later has no effect.
  • The attribute names may be in a tuple or list but preferably in a tuple since it should be immutable.
  • Subclasses will inherit the __slots__ attribute of the parent class.
class TestWithSlots(object):
      __slots__=['a', 'b']
      def __init__(self, *args, **kwargs):
                self.a = 1
                self.b = 2
                
class SubclassOfTestWithSlots(TestWithSlots):
    pass
  
if __name__ == "__main__":
     subclass = SubclassOfTestWithSlots()
     print(subclass.__slots__)
# ['a', 'b']
  • If we set an attribute that is missing from the __slots__ array it will be stored in the original __dict__ attribute of the subclassed instance.
class TestWithSlots(object):
      __slots__=['a', 'b']
      def __init__(self, *args, **kwargs):
                self.a = 1
                self.b = 2
                
class SubclassOfTestWithSlots(TestWithSlots):
    pass
  
if __name__ == "__main__":
     subclass = SubclassOfTestWithSlots()
     print(subclass.__slots__)
     subclass.x = 'c'
     print(subclass.__dict__)
     
# ['a', 'b']
# {'x': 'c'}
  • To make sure that the subclass instance has no __dict__ attribute we must add the __slots__ attribute to it too.

Conclusion

The aim of these last two articles was to demonstrate the use of special methods and conventions in the construction of a well-behaved Pythonic class and Pythonic objects.

To summarize I would like to quote the author of the original book Fluent Python:

To build Pythonic objects, observe how real Python objects behave.

Further Reading

Read more