Detailed introduction to positioning and destruction in Python threads (with examples)

This article brings you a detailed introduction to positioning and destruction in Python threads (with examples). It has certain reference value. Friends in need can refer to it. I hope it will be helpful to you.

I felt something was not right before work started, and I felt like I was going to take the blame. No, it’s the third day of work that makes me feel guilty.

We have an amazing background program that can dynamically load modules and run them in threads. In this way, the functions of the plug-in can be realized. When the module is updated, the background program itself will not exit. It will only close the thread corresponding to the module, update the code and then start it. 6 is not possible.

So I wrote a module to show off my skills, but I forgot to write the exit function, which resulted in a new thread being created every time the module was updated. Unless the program was restarted, those threads would remain alive.

This is not possible. We have to find a way to clean it up, otherwise I am afraid it will explode.

So how to clean it? All I can think of is two steps:

1. Find out the thread numbers tid that need to be cleaned up;

2. Destroy them;

Find out the thread ID

Similar to the usual troubleshooting, first check the thread status of the target process through the ps command. Because the thread name has been set by setName, normally it should I saw the corresponding thread. Directly use the following code to simulate this thread:

Python version of multi-threading

#coding: utf8
import threading
import os
import time

def tt():
    info = threading.currentThread()
    while True:
        print 'pid: ', os.getpid()
        print info.name, info.ident
        time.sleep(3)

t1 = threading.Thread(target=tt)
t1.setName('OOOOOPPPPP')
t1.setDaemon(True)
t1.start()

t2 = threading.Thread(target=tt)
t2.setName('EEEEEEEEE')
t2.setDaemon(True)
t2.start()


t1.join()
t2.join()

Output:

root@10-46-33-56:~# python t.py
pid:  5613
OOOOOPPPPP 139693508122368
pid:  5613
EEEEEEEEE 139693497632512
...

You can see that the thread name output in Python is what we set That, but the result of Ps made me doubt life:

root@10-46-33-56:~# ps -Tp 5613
  PID  SPID TTY          TIME CMD
 5613  5613 pts/2    00:00:00 python
 5613  5614 pts/2    00:00:00 python
 5613  5615 pts/2    00:00:00 python

Normally it shouldn’t be like this. I’m a little confused. Did I remember it wrong all along? Use another language version of multi-threading to test:

C version of multi-threading

#include<stdio.h>
#include<sys>
#include<sys>
#include<pthread.h>

void *test(void *name)
{    
    pid_t pid, tid;
    pid = getpid();
    tid = syscall(__NR_gettid);
    char *tname = (char *)name;
    
    // 设置线程名字
    prctl(PR_SET_NAME, tname);
    
    while(1)
    {
        printf("pid: %d, thread_id: %u, t_name: %s\n", pid, tid, tname);
        sleep(3);
    }
}

int main()
{
    pthread_t t1, t2;
    void *ret;
    pthread_create(&t1, NULL, test,  (void *)"Love_test_1");
    pthread_create(&t2, NULL, test,  (void *)"Love_test_2");
    pthread_join(t1, &ret);
    pthread_join(t2, &ret);
}</pthread.h></sys></sys></stdio.h>

Output:

root@10-46-33-56:~# gcc t.c -lpthread && ./a.out
pid: 5575, thread_id: 5577, t_name: Love_test_2
pid: 5575, thread_id: 5576, t_name: Love_test_1
pid: 5575, thread_id: 5577, t_name: Love_test_2
pid: 5575, thread_id: 5576, t_name: Love_test_1
...

Use the PS command to verify again:

root@10-46-33-56:~# ps -Tp 5575
  PID  SPID TTY          TIME CMD
 5575  5575 pts/2    00:00:00 a.out
 5575  5576 pts/2    00:00:00 Love_test_1
 5575  5577 pts/2    00:00:00 Love_test_2

This is correct. The thread name can indeed be seen through Ps!

But why can’t I see the Python one? Since the thread name is set through setName, let’s take a look at the definition:

[threading.py]
class Thread(_Verbose):
    ...
    @property
    def name(self):
        """A string used for identification purposes only.

        It has no semantics. Multiple threads may be given the same name. The
        initial name is set by the constructor.

        """
        assert self.__initialized, "Thread.__init__() not called"
        return self.__name

    @name.setter
    def name(self, name):
        assert self.__initialized, "Thread.__init__() not called"
        self.__name = str(name)
        
    def setName(self, name):
        self.name = name
    ...

Seeing here is actually just setting the properties of the Thread object, and If it has not been touched to the root, it must be invisible~

It seems that we can no longer use ps or /proc/ to externally The python thread name is searched, so we can only solve it within Python.

So the question becomes, how to get all running threads inside Python?

threading.enumerate can perfectly solve this problem! Why?

Because it is clearly stated in the doc of the following function, all active thread objects are returned, excluding terminated and unstarted ones.

[threading.py]

def enumerate():
    """Return a list of all Thread objects currently alive.

    The list includes daemonic threads, dummy thread objects created by
    current_thread(), and the main thread. It excludes terminated threads and
    threads that have not yet been started.

    """
    with _active_limbo_lock:
        return _active.values() + _limbo.values()

Because we get the Thread object, we can get the information related to the thread through this!

Please see the complete code example:

<pre class="brush:php;toolbar:false">#coding: utf8 import threading import os import time def get_thread():     pid = os.getpid()     while True:         ts = threading.enumerate()         print '------- Running threads On Pid: %d -------' % pid         for t in ts:             print t.name, t.ident         print         time.sleep(1)          def tt():     info = threading.currentThread()     pid = os.getpid()     while True:         print 'pid: {}, tid: {}, tname: {}'.format(pid, info.name, info.ident)         time.sleep(3)         return t1 = threading.Thread(target=tt) t1.setName('Thread-test1') t1.setDaemon(True) t1.start() t2 = threading.Thread(target=tt) t2.setName('Thread-test2') t2.setDaemon(True) t2.start() t3 = threading.Thread(target=get_thread) t3.setName('Checker') t3.setDaemon(True) t3.start() t1.join() t2.join() t3.join()</pre>

Output:

root@10-46-33-56:~# python t_show.py
pid: 6258, tid: Thread-test1, tname: 139907597162240
pid: 6258, tid: Thread-test2, tname: 139907586672384

------- Running threads On Pid: 6258 -------
MainThread 139907616806656
Thread-test1 139907597162240
Checker 139907576182528
Thread-test2 139907586672384

------- Running threads On Pid: 6258 -------
MainThread 139907616806656
Thread-test1 139907597162240
Checker 139907576182528
Thread-test2 139907586672384

------- Running threads On Pid: 6258 -------
MainThread 139907616806656
Thread-test1 139907597162240
Checker 139907576182528
Thread-test2 139907586672384

------- Running threads On Pid: 6258 -------
MainThread 139907616806656
Checker 139907576182528
...

The code looks a bit long, but the logic is quite simple, Thread-test1 and Thread-test2 prints out the current pid, thread id and thread name, and then exits after 3 seconds. This is to simulate the normal exit of the thread.

The Checker thread outputs all active threads in the current process through threading.enumerate every second.

It can be clearly seen that the information of Thread-test1 and Thread-test2 can be seen at the beginning. After they exit, only is left. MainThread and Checker are just themselves.

Destroy the specified thread

Since we can get the name and thread id, we can also kill the specified thread!

Suppose now Thread-test2 has gone black and gone crazy, and we need to stop it, then we can solve it in this way:

In the above code Basically, add and add the following code:

def _async_raise(tid, exctype):
    """raises the exception, performs cleanup if needed"""
    tid = ctypes.c_long(tid)
    if not inspect.isclass(exctype):
        exctype = type(exctype)
    res = ctypes.pythonapi.PyThreadState_SetAsyncExc(tid, ctypes.py_object(exctype))
    if res == 0:
        raise ValueError("invalid thread id")
    elif res != 1:
        ctypes.pythonapi.PyThreadState_SetAsyncExc(tid, None)
        raise SystemError("PyThreadState_SetAsyncExc failed")

def stop_thread(thread):
    _async_raise(thread.ident, SystemExit)

def get_thread():
    pid = os.getpid()
    while True:
        ts = threading.enumerate()
        print '------- Running threads On Pid: %d -------' % pid
        for t in ts:
            print t.name, t.ident, t.is_alive()
            if t.name == 'Thread-test2':
                print 'I am go dying! Please take care of yourself and drink more hot water!'
                stop_thread(t)
        print
        time.sleep(1)

Output

root@10-46-33-56:~# python t_show.py
pid: 6362, tid: 139901682108160, tname: Thread-test1
pid: 6362, tid: 139901671618304, tname: Thread-test2
------- Running threads On Pid: 6362 -------
MainThread 139901706389248 True
Thread-test1 139901682108160 True
Checker 139901661128448 True
Thread-test2 139901671618304 True
Thread-test2: I am go dying. Please take care of yourself and drink more hot water!

------- Running threads On Pid: 6362 -------
MainThread 139901706389248 True
Thread-test1 139901682108160 True
Checker 139901661128448 True
Thread-test2 139901671618304 True
Thread-test2: I am go dying. Please take care of yourself and drink more hot water!

pid: 6362, tid: 139901682108160, tname: Thread-test1
------- Running threads On Pid: 6362 -------
MainThread 139901706389248 True
Thread-test1 139901682108160 True
Checker 139901661128448 True
// Thread-test2 已经不在了

After a while, although we treat Thread-test2 like this, it still cares about us :Drink more hot water,

PS: Although hot water is good, eight cups is enough, so don’t be greedy.

Back to the main story, the above method is extremely crude. Why do you say that?

Because its principle is: use Python's built-in API to trigger an exception on the specified thread so that it can automatically exit;

# #Don't use this method as a last resort, as there is a certain probability of triggering indescribable problems. Remember! Don't ask me why I know...

Why is it so difficult to stop threads

Multi-threading itself is designed for collaborative concurrency under the process. It is the smallest unit of scheduling, and the threads share the process. resources, so there will be many locking mechanisms and state controls.

If you use force to kill threads, there is a high chance that unexpected bugs will occur. And the most important lock resource release may also cause unexpected problems.

We can't even kill threads directly like killing processes through signals, because kill can only achieve our expectations by dealing with processes, but it is obviously not possible to deal with threads. No matter which thread is killed, the entire process will exit!

And because of the GIL, many children think that Python's threads are implemented by Python itself and do not actually exist. Python should be able to destroy them directly, right?

However, in fact, Python’s threads are genuine threads!

What does that mean? Python threads are native threads created by the operating system through pthread. Python only uses GIL to constrain these threads to decide when to start scheduling. For example, after running a number of instructions, the GIL will be handed over. As for who wins the oiran, it depends on the operating system.

If it is a simple thread, the system actually has ways to terminate it, such as: pthread_exit,pthread_kill or pthread_cancel, for details, see: https://www.cnblogs.com/Creat...

What a pity: There is no encapsulation of these methods at the Python level! Oh my god, so angry! Maybe people think that threads should be treated gently.

How to exit a thread gently

If you want to exit a thread gently, it is almost nonsense~

Either exit after running, or set the flag bit, and check the flag bit frequently. If you need to quit, just quit.

Extension

"How to correctly terminate a running child thread": https://www.cnblogs.com/Creat...
"Don't destroy python threads roughly" ：http://xiaorui.cc/2017/02/22/...

Welcome all experts to give advice and exchange, QQ discussion group: 258498217
Please note for reprinting Source: https://segmentfault.com/a/11...

1. Column

2. Technology Stack Hodgepodge

3. Article details

#                                                                     

Lin_R                                                                                                                                                                                           posted on                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             Posted 3 days ago Python: Thread positioning and destruction

c

linux

• Python

• # 266 Reading                                                             It takes 30 minutes to read                                                                                                                             

  
  

  
  

  8                       

                                                                                                                                                                                                                                                                                                                                                                                                        BackgroundI felt something was not right before the start of work, and I felt like I was going to take the blame. No, it’s the third day of work that makes me feel guilty. We have an amazing background program that can dynamically load modules and run them in threads. In this way, the functions of the plug-in can be realized. When the module is updated, the background program itself will not exit. It will only close the thread corresponding to the module, update the code and then start it. 6 is not possible.

  So I wrote a module to show off my skills, but I forgot to write the exit function, which resulted in a new thread being created every time the module was updated. Unless the program was restarted, those threads would remain alive.

  This is not possible. We have to find a way to clean it up, otherwise I am afraid it will explode.

  So how to clean it? All I can think of is two steps:

  Find out the thread numbers tid that need to be cleaned up;

  Destroy them;

  Find out the thread ID

  1. Similar to the usual troubleshooting, first check the thread status of the target process through the ps command. Because the thread name has been set by setName, normally it should I saw the corresponding thread. Directly use the following code to simulate this thread:

  Python version of multi-threading

  2. #coding: utf8
     import threading
     import os
     import time
     
     def tt():
         info = threading.currentThread()
         while True:
             print 'pid: ', os.getpid()
             print info.name, info.ident
             time.sleep(3)
     
     t1 = threading.Thread(target=tt)
     t1.setName('OOOOOPPPPP')
     t1.setDaemon(True)
     t1.start()
     
     t2 = threading.Thread(target=tt)
     t2.setName('EEEEEEEEE')
     t2.setDaemon(True)
     t2.start()
     
     
     t1.join()
     t2.join()

     Output:

     root@10-46-33-56:~# python t.py
     pid:  5613
     OOOOOPPPPP 139693508122368
     pid:  5613
     EEEEEEEEE 139693497632512
     ...

  You can see that the thread name output in Python is what we set That, but the result of Ps made me doubt life:

  root@10-46-33-56:~# ps -Tp 5613
    PID  SPID TTY          TIME CMD
   5613  5613 pts/2    00:00:00 python
   5613  5614 pts/2    00:00:00 python
   5613  5615 pts/2    00:00:00 python

  Normally it shouldn’t be like this. I’m a little confused. Did I remember it wrong all along? Use another language version of multi-threading to test:

  C version of multi-threading

  #include<stdio.h>
  #include<sys>
  #include<sys>
  #include<pthread.h>
  
  void *test(void *name)
  {    
      pid_t pid, tid;
      pid = getpid();
      tid = syscall(__NR_gettid);
      char *tname = (char *)name;
      
      // 设置线程名字
      prctl(PR_SET_NAME, tname);
      
      while(1)
      {
          printf("pid: %d, thread_id: %u, t_name: %s\n", pid, tid, tname);
          sleep(3);
      }
  }
  
  int main()
  {
      pthread_t t1, t2;
      void *ret;
      pthread_create(&t1, NULL, test,  (void *)"Love_test_1");
      pthread_create(&t2, NULL, test,  (void *)"Love_test_2");
      pthread_join(t1, &ret);
      pthread_join(t2, &ret);
  }</pthread.h></sys></sys></stdio.h>

  Output:

  root@10-46-33-56:~# gcc t.c -lpthread && ./a.out
  pid: 5575, thread_id: 5577, t_name: Love_test_2
  pid: 5575, thread_id: 5576, t_name: Love_test_1
  pid: 5575, thread_id: 5577, t_name: Love_test_2
  pid: 5575, thread_id: 5576, t_name: Love_test_1
  ...

  Use the PS command to verify again:

  root@10-46-33-56:~# ps -Tp 5575
    PID  SPID TTY          TIME CMD
   5575  5575 pts/2    00:00:00 a.out
   5575  5576 pts/2    00:00:00 Love_test_1
   5575  5577 pts/2    00:00:00 Love_test_2

  This is correct. The thread name can indeed be seen through Ps!

  But why can’t I see the Python one? Since the thread name is set through

  setName

  , let’s take a look at the definition:

  [threading.py]
  class Thread(_Verbose):
      ...
      @property
      def name(self):
          """A string used for identification purposes only.
  
          It has no semantics. Multiple threads may be given the same name. The
          initial name is set by the constructor.
  
          """
          assert self.__initialized, "Thread.__init__() not called"
          return self.__name
  
      @name.setter
      def name(self, name):
          assert self.__initialized, "Thread.__init__() not called"
          self.__name = str(name)
          
      def setName(self, name):
          self.name = name
      ...

  Seeing here is actually just setting the properties of the

  Thread

  object, and If it has not been touched to the root, it must be invisible~

  It seems that we can no longer use

  ps

  or /proc/ to externally The python thread name is searched, so we can only solve it within Python.

  So the question becomes, how to get all running threads inside Python?

  threading.enumerate

  can perfectly solve this problem! Why?Because it is clearly stated in the doc of the following function, all active thread objects are returned, excluding terminated and unstarted ones.

  [threading.py]
  
  def enumerate():
      """Return a list of all Thread objects currently alive.
  
      The list includes daemonic threads, dummy thread objects created by
      current_thread(), and the main thread. It excludes terminated threads and
      threads that have not yet been started.
  
      """
      with _active_limbo_lock:
          return _active.values() + _limbo.values()

  Because we get the Thread object, we can get the information related to the thread through this!

  Please see the complete code example: <pre class="brush:php;toolbar:false">#coding: utf8 import threading import os import time def get_thread():     pid = os.getpid()     while True:         ts = threading.enumerate()         print '------- Running threads On Pid: %d -------' % pid         for t in ts:             print t.name, t.ident         print         time.sleep(1)          def tt():     info = threading.currentThread()     pid = os.getpid()     while True:         print 'pid: {}, tid: {}, tname: {}'.format(pid, info.name, info.ident)         time.sleep(3)         return t1 = threading.Thread(target=tt) t1.setName('Thread-test1') t1.setDaemon(True) t1.start() t2 = threading.Thread(target=tt) t2.setName('Thread-test2') t2.setDaemon(True) t2.start() t3 = threading.Thread(target=get_thread) t3.setName('Checker') t3.setDaemon(True) t3.start() t1.join() t2.join() t3.join()</pre>Output:

  root@10-46-33-56:~# python t_show.py
  pid: 6258, tid: Thread-test1, tname: 139907597162240
  pid: 6258, tid: Thread-test2, tname: 139907586672384
  
  ------- Running threads On Pid: 6258 -------
  MainThread 139907616806656
  Thread-test1 139907597162240
  Checker 139907576182528
  Thread-test2 139907586672384
  
  ------- Running threads On Pid: 6258 -------
  MainThread 139907616806656
  Thread-test1 139907597162240
  Checker 139907576182528
  Thread-test2 139907586672384
  
  ------- Running threads On Pid: 6258 -------
  MainThread 139907616806656
  Thread-test1 139907597162240
  Checker 139907576182528
  Thread-test2 139907586672384
  
  ------- Running threads On Pid: 6258 -------
  MainThread 139907616806656
  Checker 139907576182528
  ...

  The code looks a bit long, but the logic is quite simple, Thread-test1 and

  Thread-test2

  prints out the current pid, thread id and thread name, and then exits after 3 seconds. This is to simulate the normal exit of the thread.

  The

  Checker

  thread outputs all active threads in the current process through

  threading.enumerate

  every second. It can be clearly seen that the information of Thread-test1 and

  Thread-test2

  can be seen at the beginning. After they exit, only is left. MainThread and Checker are just themselves.

  Destroy the specified threadSince we can get the name and thread id, we can also kill the specified thread! Suppose now Thread-test2 has gone black and gone crazy, and we need to stop it, then we can solve it in this way: In the above code Basically, add and add the following code:

  def _async_raise(tid, exctype):
      """raises the exception, performs cleanup if needed"""
      tid = ctypes.c_long(tid)
      if not inspect.isclass(exctype):
          exctype = type(exctype)
      res = ctypes.pythonapi.PyThreadState_SetAsyncExc(tid, ctypes.py_object(exctype))
      if res == 0:
          raise ValueError("invalid thread id")
      elif res != 1:
          ctypes.pythonapi.PyThreadState_SetAsyncExc(tid, None)
          raise SystemError("PyThreadState_SetAsyncExc failed")
  
  def stop_thread(thread):
      _async_raise(thread.ident, SystemExit)
  
  def get_thread():
      pid = os.getpid()
      while True:
          ts = threading.enumerate()
          print '------- Running threads On Pid: %d -------' % pid
          for t in ts:
              print t.name, t.ident, t.is_alive()
              if t.name == 'Thread-test2':
                  print 'I am go dying! Please take care of yourself and drink more hot water!'
                  stop_thread(t)
          print
          time.sleep(1)

  Output

  root@10-46-33-56:~# python t_show.py
  pid: 6362, tid: 139901682108160, tname: Thread-test1
  pid: 6362, tid: 139901671618304, tname: Thread-test2
  ------- Running threads On Pid: 6362 -------
  MainThread 139901706389248 True
  Thread-test1 139901682108160 True
  Checker 139901661128448 True
  Thread-test2 139901671618304 True
  Thread-test2: I am go dying. Please take care of yourself and drink more hot water!
  
  ------- Running threads On Pid: 6362 -------
  MainThread 139901706389248 True
  Thread-test1 139901682108160 True
  Checker 139901661128448 True
  Thread-test2 139901671618304 True
  Thread-test2: I am go dying. Please take care of yourself and drink more hot water!
  
  pid: 6362, tid: 139901682108160, tname: Thread-test1
  ------- Running threads On Pid: 6362 -------
  MainThread 139901706389248 True
  Thread-test1 139901682108160 True
  Checker 139901661128448 True
  // Thread-test2 已经不在了

  After a while, although we treat

  Thread-test2

  like this, it still cares about us :Drink more hot water,

  PS: Although hot water is good, eight cups is enough, so don’t be greedy.

  Back to the main story, the above method is extremely crude. Why do you say that?

  Because its principle is: use Python's built-in API to trigger an exception on the specified thread so that it can automatically exit;

  

  # #Don't use this method as a last resort, as there is a certain probability of triggering indescribable problems. Remember! Don't ask me why I know...
  Why is it so difficult to stop threads
  Multi-threading itself is designed for collaborative concurrency under the process. It is the smallest unit of scheduling, and the threads share the process. resources, so there will be many locking mechanisms and state controls.
  If you use force to kill threads, there is a high chance that unexpected bugs will occur. And the most important lock resource release may also cause unexpected problems.
  We can't even kill the thread directly like killing the process through a signal, because kill can only achieve our expectations by dealing with the process, but it is obviously not possible to deal with the thread. No matter which thread is killed, the entire process will exit!
  And because of the GIL, many children think that Python's threads are implemented by Python itself and do not actually exist. Python should be able to destroy them directly, right?
  However, in fact, Python’s threads are genuine threads!
  What does that mean? Python threads are native threads created by the operating system through pthread. Python only uses GIL to constrain these threads to decide when to start scheduling. For example, after running a number of instructions, the GIL will be handed over. As for who wins the oiran, it depends on the operating system.
  If it is a simple thread, the system actually has ways to terminate it, such as:

  pthread_exit,pthread_kill or pthread_cancel, for details, see: https://www.cnblogs.com/Creat...

  What a pity: There is no encapsulation of these methods at the Python level! Oh my god, so angry! Maybe people think that threads should be treated gently.
  How to exit a thread gently
  If you want to exit a thread gently, it is almost nonsense~
  Either exit after running, or set the flag bit, and check the flag bit frequently. If you need to quit, just quit.
  Extension
  "How to correctly terminate a running child thread": https://www.cnblogs.com/Creat...

  "Don't destroy python threads roughly" ：http://xiaorui.cc/2017/02/22/...
  

  Welcome all experts to give advice and exchange, QQ discussion group: 258498217
  Please note for reprinting Ming source: https://segmentfault.com/a/11...

  • All rights reserved

  • report

  
  

  
  

  
  

  If you find my article useful to you, please feel free to appreciate it

  You may feel Interested

  
  

  
  

  ##2 comments

                                                                                             Time sorting

  
  

  
  

  ##                  

  
  

  

  
  

                                                                                                             
  

  · 1 day ago           

  If it were me, I might kill -9, I would rather kill a thousand by mistake than let one go, hehe

                                                                            Thumbs up                                                                                                                                   reply                                                

                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
  

  ##No way~ -9 The process is all dead~

                                                                                                                                                                                                                                                                                                                                                                                                                                                                         —                                                                                                                                                                                                                                                                                        author · 1 day ago                                                                                                    Add replyLoading...Show more comments

The above is the detailed content of Detailed introduction to positioning and destruction in Python threads (with examples). For more information, please follow other related articles on the PHP Chinese website!