Home > Article > Backend Development > Let’s dive into the internals of Python classes
Let’s dive into the internals of Python classes
- coldplay.xixiforward
- 2020-11-18 17:16:001642browse
python video tutorialThe column introduces the internals of Python classes.
This article will talk with you about some concepts and implementation principles behind classes and objects in Python 3.8. It mainly tries to explain the properties of Python classes and objects. Optimization support for storage, functions and methods, descriptors, object memory usage, and related issues such as inheritance and attribute lookup.
Let's start with a simple example:
class Employee:
outsource = False
def __init__(self, department, name):
self.department = department
self.name = name @property
def inservice(self):
return self.department is not None
def __repr__(self):
return f"<Employee: {self.department}-{self.name}>"employee = Employee('IT', 'bobo')复制代码
employee The object is an instance of the Employee class and it has two propertiesdepartment and name, whose values belong to this instance. outsource is a class attribute, the owner is the class, and all instance objects of the class share this attribute value, which is consistent with other object-oriented languages.
Changing a class variable will affect all instance objects of the class:
>>> e1 = Employee('IT', 'bobo')>>> e2 = Employee('HR', 'cici')>>> e1.outsource, e2.outsource
(False, False)>>> Employee.outsource = True>>> e1.outsource, e2.outsource>>> (True, True)复制代码
This is limited to changes from the class, when we change the class variable from the instance:
>>> e1 = Employee('IT', 'bobo')>>> e2 = Employee('HR', 'cici')>>> e1.outsource, e2.outsource
(False, False)>>> e1.outsource = True>>> e1.outsource, e2.outsource
(True, False)复制代码
Yes Yes, when you try to modify a class variable from an instance object, Python does not change the class variable value of the class, but creates an instance property with the same name, which is very correct and safe. Instance variables are given priority over class variables when searching for property values, as explained in detail in the Inheritance and Property Lookup section.
It is important to note that when the type of class variables is mutable type, you change them from the instance object:
>>> class S:... L = [1, 2] ...>>> s1, s2 = S(), S()>>> s1.L, s2.L ([1, 2], [1, 2])>>> t1.L.append(3)>>> t1.L, s2.L ([1, 2, 3], [1, 2, 3])复制代码
Good practice The method is to avoid such a design as much as possible.
Storage of attributes
In this section, let’s take a look at how class attributes, methods and instance attributes in Python are associated and stored.
Instance attributes
In Python, all instance attributes are stored in the __dict__ dictionary, which is a regular dict. For instance attributes The maintenance is to obtain and modify from this dictionary, which is completely open to developers.
>>> e = Employee('IT', 'bobo')>>> e.__dict__
{'department': 'IT', 'name': 'bobo'}>>> type(e.__dict__)dict>>> e.name is e.__dict__['name']True>>> e.__dict__['department'] = 'HR'>>> e.department'HR'复制代码
Because instance attributes are stored in a dictionary, we can easily add or delete fields to the object at any time:
>>> e.age = 30 # 并没有定义 age 属性>>> e.age30>>> e.__dict__
{'department': 'IT', 'name': 'bobo', 'age': 30}>>> del e.age>>> e.__dict__
{'department': 'IT', 'name': 'd'}复制代码
We can also instantiate an object from the dictionary, Or restore the instance by saving the instance's __dict__.
>>> def new_employee_from(d):... instance = object.__new__(Employee)... instance.__dict__.update(d)... return instance
...>>> e1 = new_employee_from({'department': 'IT', 'name': 'bobo'})>>> e1
<Employee: IT-bobo>>>> state = e1.__dict__.copy()>>> del e1>>> e2 = new_employee_from(state)>>> e2>>> <Employee: IT-bobo>复制代码
Because __dict__ is completely open, we can add any hashable immutable key to it, such as numbers:
>>> e.__dict__[1] = 1>>> e.__dict__
{'department': 'IT', 'name': 'bobo', 1: 1}复制代码 These non-string fields cannot be accessed through instance objects. In order to ensure that such a situation does not occur, it is generally best not to write directly to __dict__ unless necessary. Don't even manipulate __dict__ directly.
So there is a saying that Python is a "consenting adults language".
This dynamic implementation makes our code very flexible and very convenient in many cases, but it also comes with storage and performance overhead. Therefore, Python also provides another mechanism (__slots__) to abandon the use of __dict__ to save memory and improve performance. See the __slots__ section for details.
Class attributes
Similarly, class attributes are also stored in the __dict__ dictionary of the class:
>>> Employee.__dict__
mappingproxy({'__module__': '__main__', 'outsource': True, '__init__': <function __main__.Employee.__init__(self, department, name)>, 'inservice': <property at 0x108419ea0>, '__repr__': <function __main__.Employee.__repr__(self)>, '__str__': <function __main__.Employee.__str__(self)>, '__dict__': <attribute '__dict__' of 'Employee' objects>, '__weakref__': <attribute '__weakref__' of 'Employee' objects>, '__doc__': None}>>> type(Employee.__dict__)
mappingproxy复制代码
is different from the "open" of the instance dictionary , the dictionary used by the class attribute is a MappingProxyType object, which is a dictionary that cannot be setattr. This means that it is read-only for developers, and its purpose is to ensure that the keys of class attributes are all strings to simplify and speed up the search logic of new class attributes and __mro__.
>>> Employee.__dict__['outsource'] = FalseTypeError: 'mappingproxy' object does not support item assignment复制代码
Because all methods belong to a class, they are also stored in the class dictionary. From the above example, you can see the existing __init__ and __repr__ method. We can add a few more to verify:
class Employee: # ... @staticmethod def soo(): pass @classmethod def coo(cls): pass def foo(self): pass复制代码
>>> Employee.__dict__
mappingproxy({'__module__': '__main__', 'outsource': False, '__init__': <function __main__.Employee.__init__(self, department, name)>, '__repr__': <function __main__.Employee.__repr__(self)>, 'inservice': <property at 0x108419ea0>, 'soo': <staticmethod at 0x1066ce588>, 'coo': <classmethod at 0x1066ce828>, 'foo': <function __main__.Employee.foo(self)>, '__dict__': <attribute '__dict__' of 'Employee' objects>, '__weakref__': <attribute '__weakref__' of 'Employee' objects>, '__doc__': None})复制代码
Inheritance and attribute lookup
So far, we already know that all attributes and methods are stored in two __dict__ In the dictionary, now let's take a look at how Python performs attribute lookup.
In Python 3, all classes implicitly inherit from object, so there is always an inheritance relationship, and Python supports multiple inheritance:
>>> class A:... pass...>>> class B:... pass...>>> class C(B):... pass...>>> class D(A, C):... pass...>>> D.mro() [<class '__main__.D'>, <class '__main__.A'>, <class '__main__.C'>, <class '__main__.B'>, <class 'object'>]复制代码
mro() is a special method that returns the linear parsing order of the class.
The default behavior of attribute access is to get, set or delete attributes from the object's dictionary. For example, the simple description of the lookup for e.f is:
e.f的查找顺序会从e.__dict__['f']开始，然后是type(e).__dict__['f']，接下来依次查找type(e)的基类（__mro__顺序，不包括元类）。 如果找到的值是定义了某个描述器方法的对象，则 Python 可能会重载默认行为并转而发起调用描述器方法。这具体发生在优先级链的哪个环节则要根据所定义的描述器方法及其被调用的方式来决定。
所以，要理解查找的顺序，你必须要先了解描述器协议。
简单总结，有两种描述器类型：数据描述器和和非数据描述器。
如果一个对象除了定义
__get__()之外还定义了__set__()或__delete__()，则它会被视为数据描述器。仅定义了__get__()的描述器称为非数据描述器（它们通常被用于方法，但也可以有其他用途)
由于函数只实现 __get__，所以它们是非数据描述器。
Python 的对象属性查找顺序如下：
- 类和父类字典的数据描述器
- 实例字典
- 类和父类字典中的非数据描述器
请记住，无论你的类有多少个继承级别，该类对象的实例字典总是存储了所有的实例变量，这也是 super 的意义之一。
下面我们尝试用伪代码来描述查找顺序：
def get_attribute(obj, name): class_definition = obj.__class__ descriptor = None for cls in class_definition.mro(): if name in cls.__dict__: descriptor = cls.__dict__[name] break if hasattr(descriptor, '__set__'): return descriptor, 'data descriptor' if name in obj.__dict__: return obj.__dict__[name], 'instance attribute' if descriptor is not None: return descriptor, 'non-data descriptor' else: raise AttributeError复制代码
>>> e = Employee('IT', 'bobo')>>> get_attribute(e, 'outsource')
(False, 'non-data descriptor')>>> e.outsource = True>>> get_attribute(e, 'outsource')
(True, 'instance attribute')>>> get_attribute(e, 'name')
('bobo', 'instance attribute')>>> get_attribute(e, 'inservice')
(<property at 0x10c966d10>, 'data descriptor')>>> get_attribute(e, 'foo')
(<function __main__.Employee.foo(self)>, 'non-data descriptor')复制代码
由于这样的优先级顺序，所以实例是不能重载类的数据描述器属性的，比如 property 属性：
>>> class Manager(Employee):... def __init__(self, *arg):... self.inservice = True... super().__init__(*arg)
...>>> m = Manager("HR", "cici")
AttributeError: can't set attribute复制代码
发起描述器调用
上面讲到，在查找属性时，如果找到的值是定义了某个描述器方法的对象，则 Python 可能会重载默认行为并转而发起描述器方法调用。
描述器的作用就是绑定对象属性，我们假设 a 是一个实现了描述器协议的对象，对 e.a 发起描述器调用有以下几种情况：
- 直接调用：用户级的代码直接调用
e.__get__(a)，不常用 - 实例绑定：绑定到一个实例，
e.a会被转换为调用:type(e).__dict__['a'].__get__(e, type(e)) - 类绑定：绑定到一个类，
E.a会被转换为调用:E.__dict__['a'].__get__(None, E)
在继承关系中进行绑定时，会根据以上情况和 __mro__ 顺序来发起链式调用。
函数与方法
我们知道方法是属于特定类的函数，唯一的不同(如果可以算是不同的话)是方法的第一个参数往往是为类或实例对象保留的，在 Python 中，我们约定为 cls 或 self, 当然你也可以取任何名字如 this(只是最好不要这样做)。
上一节我们知道，函数实现了 __get__() 方法的对象，所以它们是非数据描述器。在 Python 访问(调用)方法支持中正是通过调用 __get__() 将调用的函数绑定成方法的。
在纯 Python 中，它的工作方式如下(示例来自描述器使用指南):
class Function: def __get__(self, obj, objtype=None): if obj is None: return self return types.MethodType(self, obj) # 将函数绑定为方法复制代码
在 Python 2 中，有两种方法: unbound method 和 bound method，在 Python 3 中只有后者。
bound method 与它们绑定的类或实例数据相关联：
>>> Employee.coo
<bound method Employee.coo of <class '__main__.Employee'>>
>>> Employee.foo<function __main__.Employee.foo(self)>
>>> e = Employee('IT', 'bobo')
>>> e.foo<bound method Employee.foo of <Employee: IT-bobo>>复制代码
我们可以从方法来访问实例与类：
>>> e.foo.__self__ <Employee: IT-bobo>>>> e.foo.__self__.__class__ __main__.Employee复制代码
借助描述符协议，我们可以在类的外部作用域手动绑定一个函数到方法，以访问类或实例中的数据，我将以这个示例来解释当你的对象访问(调用)类字典中存储的函数时将其绑定成方法(执行)的过程：
现有以下函数：
>>> def f1(self):... if isinstance(self, type):... return self.outsource... return self.name ...>>> bound_f1 = f1.__get__(e, Employee) # or bound_f1 = f1.__get__(e)>>> bound_f1 <bound method f1 of <Employee: IT-bobo>>>>> bound_f1.__self__ <Employee: IT-bobo>>>> bound_f1()'bobo'复制代码
总结一下：当我们调用 e.foo() 时，首先从 Employee.__dict__['foo'] 中得到 foo 函数，在调用该函数的 foo 方法 foo.__get__(e) 将其转换成方法，然后执行 foo() 获得结果。这就完成了 e.foo() -> f(e) 的过程。
如果你对我的解释感到疑惑，我建议你可以阅读官方的描述器使用指南以进一步了解描述器协议，在该文的函数和方法和静态方法和类方法一节中详细了解函数绑定为方法的过程。同时在 Python 类一文的方法对象一节中也有相关的解释。
__slots__
Python 的对象属性值都是采用字典存储的，当我们处理数成千上万甚至更多的实例时，内存消耗可能是一个问题，因为字典哈希表的实现，总是为每个实例创建了大量的内存。所以 Python 提供了一种 __slots__ 的方式来禁用实例使用 __dict__，以优化此问题。
通过 __slots__ 来指定属性后，会将属性的存储从实例的 __dict__ 改为类的 __dict__ 中：
class Test:
__slots__ = ('a', 'b') def __init__(self, a, b):
self.a = a
self.b = b复制代码
>>> t = Test(1, 2)>>> t.__dict__
AttributeError: 'Test' object has no attribute '__dict__'>>> Test.__dict__
mappingproxy({'__module__': '__main__', '__slots__': ('a', 'b'), '__init__': <function __main__.Test.__init__(self, a, b)>, 'a': <member 'a' of 'Test' objects>, 'b': <member 'b' of 'Test' objects>, '__doc__': None})复制代码
关于 __slots__ 我之前专门写过一篇文章分享过，感兴趣的同学请移步理解 Python 类属性 __slots__ 一文。
补充
__getattribute__ 和 __getattr__
也许你还有疑问，那函数的 __get__ 方法是怎么被调用的呢，这中间过程是什么样的？
在 Python 中 一切皆对象，所有对象都有一个默认的方法 __getattribute__(self, name)。
该方法会在我们使用 . 访问 obj 的属性时会自动调用，为了防止递归调用，它总是实现为从基类 object 中获取 object.__getattribute__(self, name), 该方法大部分情况下会默认从 self 的 __dict__ 字典中查找 name(除了特殊方法的查找)。
话外：如果该类还实现了
__getattr__，则只有__getattribute__显式地调用或是引发了AttributeError异常后才会被调用。__getattr__由开发者自己实现，应当返回属性值或引发AttributeError异常。
而描述器正是由 __getattribute__() 方法调用，其大致逻辑为：
def __getattribute__(self, key): v = object.__getattribute__(self, key) if hasattr(v, '__get__'): return v.__get__(self) return v复制代码
请注意：重写
__getattribute__()会阻止描述器的自动调用。
函数属性
函数也是 Python function 对象，所以一样，它也具有任意属性，这有时候是有用的，比如实现一个简单的函数调用跟踪装饰器：
def calltracker(func): @wraps(func) def wrapper(*args, **kwargs): wrapper.calls += 1 return func(*args, **kwargs) wrapper.calls = 0 return wrapper@calltrackerdef f(): return 'f called'复制代码
>>> f.calls0>>> f()'f called'>>> f.calls1复制代码
相关免费学习推荐：python视频教程
The above is the detailed content of Let’s dive into the internals of Python classes. For more information, please follow other related articles on the PHP Chinese website!