Cleverly solve Python multi-thread deadlock problem

【Related learning recommendations: python video】

Today is the 25th article of Python Special Topic. Let’s talk about the problem of deadlock in multi-threaded development.

Deadlock

The principle of deadlock is very simple , can be described in one sentence. That is, when multiple threads access multiple locks, different locks are held by different threads, and they are all waiting for other threads to release the locks, so they fall into a permanent wait. For example, thread A holds lock number 1 and is waiting for lock number 2, and thread B holds lock number 2 and is waiting for lock number 1. Then they will never wait until the day of execution. This situation is called a deadlock.

There is a famous problem about deadlock called Philosophers Dining problem. There are 5 philosophers sitting together, and each of them needs to get two forks before they can eat. If they pick up their left-hand forks at the same time, they will wait forever for the right-hand fork to be released. This will lead to eternal waiting, and these philosophers will starve to death.

Cleverly solve Python multi-thread deadlock problem

This is a very vivid model, because in computer concurrency scenarios, the number of some resources is often limited. It is very likely that multiple threads will preempt. If not handled well, everyone will obtain a resource and then wait for another resource.

There are many solutions to the deadlock problem. Here we introduce a simpler one, which is to number these locks. We stipulate that when a thread needs to hold multiple locks at the same time, must access these locks in ascending order of serial numbers . We can easily achieve this through context managers.

Context Manager

First let’s briefly introduce context management In fact, we often use context managers. For example, the with statement we often use is a classic use of context managers. When we open a file through the with statement, it will automatically handle the closing of the file after reading and the handling of exceptions thrown, which can save us a lot of code.

Similarly, we can also define a context processor ourselves. It is actually very simple. We only need to implement the two functions __enter__ and __exit__. The __enter__ function is used to implement operations and processing before entering the resource, so obviously the __exit__ function corresponds to the processing logic after the use of the resource is completed or an exception occurs. With these two functions, we have our own context processing class.

Let’s take a look at an example:

class Sample:    def __enter__(self):        print('enter resources')        return self        def __exit__(self, exc_type, exc_val, exc_tb):        print('exit')        # print(exc_type)        # print(exc_val)        # print(exc_tb)    def doSomething(self):        a = 1/1        return adef getSample():    return Sample()if __name__ == '__main__':    with getSample() as sample:        print('do something')        sample.doSomething()复制代码

When we run this code, the results printed on the screen are consistent with our expectations.

Cleverly solve Python multi-thread deadlock problem

If we look at the __exit__ function, we will find that it has 4 parameters, and the three parameters after correspond to is the case where an exception is thrown. type corresponds to the type of exception, val corresponds to the output value when the exception occurs, and trace corresponds to the running stack when the exception is thrown. This information is often used when we troubleshoot exceptions. Through these three fields, we can customize processing of possible exceptions according to our needs.

Implementing a context manager does not necessarily have to be implemented through a class. Python also provides context management annotations. By using annotations, we can easily implement context management. Let's also look at an example:

import timefrom contextlib import contextmanager@contextmanagerdef timethis(label):    start = time.time()    try:        yield    finally:        end = time.time()        print('{}: {}'.format(label, end - start))                with timethis('timer'):    pass复制代码

In this method, the part before yield is equivalent to the __enter__ function, and the part after yield is equivalent to __exit__. If an exception occurs, it will be thrown in the try statement, then we can write except to handle the exception.

避免死锁

了解了上下文管理器之后，我们要做的就是在lock的外面包装一层，使得我们在获取和释放锁的时候可以根据我们的需要，对锁进行排序，按照升序的顺序进行持有。

这段代码源于Python的著名进阶书籍《Python cookbook》，非常经典：

from contextlib import contextmanager# 用来存储local的数据_local = threading.local()@contextmanagerdef acquire(*locks): # 对锁按照id进行排序    locks = sorted(locks, key=lambda x: id(x))    # 如果已经持有锁当中的序号有比当前更大的，说明策略失败    acquired = getattr(_local,'acquired',[])    if acquired and max(id(lock) for lock in acquired) >= id(locks[0]):        raise RuntimeError('Lock Order Violation')    # 获取所有锁    acquired.extend(locks)    _local.acquired = acquired    try:        for lock in locks:            lock.acquire()        yield    finally:        # 倒叙释放        for lock in reversed(locks):            lock.release()        del acquired[-len(locks):]复制代码

这段代码写得非常漂亮，可读性很高，逻辑我们都应该能看懂，但是有一个小问题是这里用到了threading.local这个组件。

它是一个多线程场景当中的共享变量，虽然说是共享的，但是对于每个线程来说读取到的值都是独立的。听起来有些难以理解，其实我们可以将它理解成一个dict，dict的key是每一个线程的id，value是一个存储数据的dict。每个线程在访问local变量的时候，都相当于先通过线程id获取了一个独立的dict，再对这个dict进行的操作。

看起来我们在使用的时候直接使用了_local，这是因为通过线程id先进行查询的步骤在其中封装了。不明就里的话可能会觉得有些难以理解。

我们再来看下这个acquire的使用：

x_lock = threading.Lock()y_lock = threading.Lock()def thread_1():    while True:        with acquire(x_lock, y_lock):            print('Thread-1')def thread_2():    while True:        with acquire(y_lock, x_lock):            print('Thread-2')t1 = threading.Thread(target=thread_1)t1.start()t2 = threading.Thread(target=thread_2)t2.start()复制代码

运行一下会发现没有出现死锁的情况，但如果我们把代码稍加调整，写成这样，那么就会触发异常了。

def thread_1():    while True:        with acquire(x_lock):            with acquire(y_lock):             print('Thread-1')def thread_2():    while True:        with acquire(y_lock):            with acquire(x_lock):             print('Thread-1')复制代码

因为我们把锁写成了层次结构，这样就没办法进行排序保证持有的有序性了，那么就会触发我们代码当中定义的异常。

最后我们再来看下哲学家就餐问题，通过我们自己实现的acquire函数我们可以非常方便地解决他们死锁吃不了饭的问题。

import threadingdef philosopher(left, right):    while True:        with acquire(left,right):             print(threading.currentThread(), 'eating')# 叉子的数量NSTICKS = 5chopsticks = [threading.Lock() for n in range(NSTICKS)]for n in range(NSTICKS):    t = threading.Thread(target=philosopher,                         args=(chopsticks[n],chopsticks[(n+1) % NSTICKS]))    t.start()复制代码

总结

关于死锁的问题，对锁进行排序只是其中的一种解决方案，除此之外还有很多解决死锁的模型。比如我们可以让线程在尝试持有新的锁失败的时候主动放弃所有目前已经持有的锁，比如我们可以设置机制检测死锁的发生并对其进行处理等等。发散出去其实有很多种方法，这些方法起作用的原理各不相同，其中涉及大量操作系统的基础概念和知识，感兴趣的同学可以深入研究一下这个部分，一定会对操作系统以及锁的使用有一个深刻的认识。

相关学习推荐：编程视频

The above is the detailed content of Cleverly solve Python multi-thread deadlock problem. For more information, please follow other related articles on the PHP Chinese website!