Python development--detailed explanation of processes, threads, and coroutines

What is a process?

A program cannot run alone. Only when the program is loaded into memory and the system allocates resources to it can it run. This executed program is called a process. The difference between a program and a process is that a program is a collection of instructions, which is a static description text of the process; a process is an execution activity of the program and is a dynamic concept.

What is a thread?

Thread is the smallest unit that the operating system can perform operation scheduling. It is included in the process and is the actual operating unit in the process. A thread refers to a single sequential control flow in a process. Multiple threads can run concurrently in a process, and each thread performs different tasks in parallel.

What is the difference between process and thread?

Threads share memory space, and the memory of the process is independent.

Threads of the same process can communicate directly, but two processes must communicate with each other through an intermediate agent.

Creating a new thread is very simple. Creating a new process requires cloning its parent process.

A thread can control and operate other threads in the same process, but a process can only operate child processes.

Python GIL (Global Interpreter Lock)

No matter how many threads are opened and how many CPUs there are, python only allows one thread at the same time when executing.

Python threading module

Direct call

1. import threading,time
3. def run_num(num):
4. ## """
5. Define the function to be run by the thread
6. ## :param num:
7. :
   return:
## """
print("
9. running on number:%s
   "%num)
   ## time.sleep(3)
11. ##if
__name__ == '__main__':
12. # Generate a thread instance t1
13. ## t1 = threading.Thread(target=run_num,args=(1,) )
14. # Generate a thread instance t2
15. t2 = threading.Thread(target =run_num,args=(2,))
16. # Start thread t1
17. t1.start()
18. Start thread t2
19. ## t2.start()
20. # Get the thread name
21. print(t1.getName())
22. print(t2.getName())
23. Output:
24. running on number:1
25. running on number:2
26. Thread-1
27. Thread-2
28. Inherited calling
import threading,time

2. class MyThread(threading.Thread):
3. ## def __init__(self,num :
4. ## num
5. # Define the function to be run by each thread. The function name must be run
6. def run(self):
## print("
7. running on number:%s
   "%self.num)
8. time.sleep(3)
9. ##if
   __name__ == '__main__':
## t1 = MyThread(1)
11. t2 = MyThread(2)
12. ## t1.start()
13. t2.start()
14. Output:
15. ##running on number :1
16. running on number:2
##Join and Daemon
17. Join The function of Join is to block the main process and cannot execute the program after the join.
In the case of multi-threads and multiple joins, the join methods of each thread are executed sequentially. The execution of the previous thread is completed before the subsequent thread can be executed.
18. When there are no parameters, wait for the thread to end before executing subsequent programs. After setting the parameters, wait for the time set by the thread and then execute the subsequent main process, regardless of whether the thread ends.
import threading,time
##class
MyThread(threading.Thread):
1. ## def __init__(self,num):
3.                                                                                                                                                                                                                       
   # Define the function to be run by each thread. The function name must be run
## def run(self):
4. ##    print("
running on number:%s
5. "%self.num)
6. time.sleep(3)
## print("
7. thread:%s
   "%self.num)
##if
9. __name__ == '__main__':
   t1 = MyThread(1)
10. ## t2 = MyThread(2)
11. t1.start()
12. t1.join()
13. t2.start()
14. ## t2.join()
15. Output:
16. running on number:1
17. thread:1
18. ##running on number:2
##thread:2
19. The effect of setting parameters is as follows:
1. if
   __name__ == '__main__':
   ## t1 = MyThread(1)
2. t2 = MyThread(2)
3. ## t1.start()
4. ## t1.join(2)
5. t2.start()
6. ## t2.join()
7. Output:
8. ##running on number:1
9. running on number:2
##thread:1
10. ##thread:2
11. Daemon
By default, the main thread will wait for the end of all child threads when exiting. If you want the main thread not to wait for the child threads, but to automatically end all child threads when exiting, you need to set the child threads as background threads (daemon). The method is to call the setDaemon() method of the thread class.
12. import time,threading
1. ##def run(n):
2. print("
   %s
".center(20,"
3. *
   ") %n)
4. time.sleep(2)## print("done
   ".center(20,"
*
5. "))
6. ##def main():
7. for
   i
   in
range(5):
8. ##   t = threading.Thread(target=run,args=(i,))
9. t . start()
10. t.join(1)
## print("
11. starting thread
    ",t.getName())
12. ##m = threading.Thread( target=main,args=())
13. # Set the main thread to the Daemon thread, which serves as the daemon thread of the main thread of the program. When the main thread exits, The m thread will also exit, and other sub-threads started by m will exit at the same time, regardless of whether the execution is completed
14. m.setDaemon(True)
15. m.start()
17. m.join(3)
18. ##print("main thread done". center(20,"*"))
19. Output:
20. *********0**********
21. starting thread Thread-2
22. *********1*********
23. ********done********
24. starting thread Thread-3
25. **********2************
26. **main thread done **
Thread lock (mutex lock Mutex)
Multiple threads can be started under one process, and multiple threads share the memory space of the parent process. This means that each thread can access the same data. At this time, if two threads want to modify the same data at the same time, a thread lock is required.
1. import time,threading
3. ##def addNum():
4. #Get this global variable in each thread
5. global num
6. print("
   --get num:",num)
7. time.sleep(1)
8. Perform -1 operation on this public variable
9. num -= 1
10. # Set a shared variable
11. num = 100
12. thread_list = []
13. ##for
    i in range(100):
    ## t = threading.Thread(target=addNum)
14. t.start()
15. ## thread_list.append(t)
16. # Wait for all threads to finish executing
17. ##for
    t
in
18. thread_list:
    t.join()
##print("
20. final num:
    ",num)
21. Locked versionLock blocks other threads from sharing Resource access, and the same thread can only acquire once. If more than once, a deadlock will occur and the program cannot continue to execute.

import time,threading

1. ##def addNum():
3. #Get this global variable in each thread
4. global num
5. print("
   --get num:",num)
6. time.sleep(1)
7. # Lock before modifying data
8. ##lock.acquire()
9. # Perform -1 operation on this public variable
10. num -= 1
11. # Release after modification
12. ##lock.release()
13. # Set a shared variable
##num = 100
thread_list = []
# Generate global lock
17. lock
    = threading.Lock()
##for
18. i
    in range (100):## t = threading.Thread(target=addNum)
19. t.start()
20. thread_list.append(t)
21. # Wait for all Thread execution completed
22. for
t
23. in
    thread_list: t.join()
25. ##print("
    final num:
    ",num)
26. GIL VS LockGIL guarantees that only one thread can execute at the same time. Lock is a user-level lock and has nothing to do with GIL.
RLock (recursive lock)
Rlock is allowed to be acquired multiple times in the same thread. The thread's release of shared resources requires the release of all locks. That is, n times of acquire require n times of release.



def run1():
1. ## print("grab the first part data")
2. lock.acquire()
3. global num
4. ## num += 1
5. ##lock
   .release()
6. return num
7. def run2():
8. ## print("
grab the second part data
9. ")
10. ​
    lock.acquire()
## ​ global num2
11. num2 += 1
12. lock
.release()
14. return
    num2
16. ##def run3():
17. ##​
    lock.acquire()
18. res = run1()
19. print ("
    between run1 and run2".center(50,"*"))
## res2 = run2 ()
21. lock
    .release()
    ## print (res,res2)
23. ##if
__name__ == '__main__':
24. num,num2 = 0,0
25. num,num2 = threading.RLock()
26. for i
    in
range(10):
27. t = threading.Thread(target=run3)
28. # t.start()
29. ##while
threading.active_count() != 1:
31. print(threading.active_count())
#else
32. :
33. print("
    all threads done".center(50,"*"))
34. Print(num,num2)The main difference between these two locks is that RLock allows to be acquired multiple times in the same thread. Lock does not allow this to happen. Note that if you use RLock, acquire and release must appear in pairs, that is, acquire is called n times, and release must be called n times to truly release the occupied lock.
Semaphore (Semaphore)
35. The mutex only allows one thread to change data at the same time, while Semaphore allows a certain number of threads to change data at the same time. For example, if the ticket office has 3 windows, then the maximum Only 3 people are allowed to buy tickets at the same time, and those at the back can only buy tickets after the people at any window in front leave.
import threading,time
1. ##def run(n):
2. semaphore.acquire()
3. ## time.sleep( 1)
4. ## print("run the thread:%s"%n)
5. semaphore.release()
#if
7. __name__ == '__main__':
##​ # Allows up to 5 threads to run at the same time
9. semaphore = threading.BoundedSemaphore(5)
##for
10. i
    in range(20):
11. t = threading.Thread(target=run,args=(i,))
12. t.start ()
14. ##while threading.active_count() != 1:
15. ​ # print(threading.active_count())
16. ## pass
17. else:
## print("
18. all threads done
    ".center(50,"*"))
    Timer (timer)
Timer calls a function at a certain interval , if you want to call a function every once in a while, you must set the Timer again in the function called by the Timer. Timer is a derived class of Thread.
import threading
2. ##def hello( ):
3. print("
hello,world!
4. ")
   # Execute the hello function after delay 5 seconds
5. t = threading.Timer(5,hello)
6. t.start()
7. Event
Python provides the Event object for inter-thread communication, which has thread settings If the signal flag is false, the thread waits for the instruction signal to be set to true by other threads. The Event object implements a simple thread communication mechanism. It provides setting signals, clearing signals, waiting, etc. for realizing communication between threads.

Set signal

1. Use the set() method of Event to set the signal flag inside the Event object to true. The Event object provides the isSet() method to determine the state of its internal signal flag. When the set() method of the event object is used, the isSet() method returns true.

Clear the signal

1. Use the clear() method of Event to clear the signal flag inside the Event object, that is, set it to false. When using After Event's clear() method, the isSet() method returns false.

Waiting

1. Event's wait() method will only execute and complete the return quickly when the internal signal is true. When the internal signal flag of the Event object is false, the wait() method waits for it to be true before returning.

Use Event to realize the interaction between two or more threads. Let's take the traffic light as an example. That is, start a thread to act as a traffic light, generate several threads to act as vehicles, and the vehicles will stop according to red and green. rule.
import threading,time,random
2. # #def light():
3. if
not
4. event
   .isSet():
   # Every
.
5.       while True:
6. ##                               ## count < 5:
7. ##      print("\033[42;1m--green light on--\033[0m".center(50,"*"))
8.         elif count < 8:
9.             print("\033[43;1m--yellow light on--\033[0m".center(50,"*"))
10.         elif count < 13:
11.             ifevent.isSet():
12.                 event.clear()
13.             print("\033[41;1m--red light on--\033[0m".center(50,"*"))
14.         else:
15.             count = 0
16.             event.set()
17.         time.sleep(1)
18.         count += 1
21. def car(n):
22.     while 1:
23.         time.sleep(random.randrange(10))
24.         ifevent.isSet():
25.             print("car %s is running..."%n)
26.         else:
27.             print("car %s is waiting for the red light..."%n)
29. if __name__ == "__main__":
30.     event = threading.Event()
31.     Light = threading.Thread(target=light,)
32.     Light.start()
34.     for i in range(3):
35.         t = threading.Thread(target=car,args=(i,))
36.         t.start()

queue queue

Queue in Python is the most commonly used form of exchanging data between threads. The Queue module is a module that provides queue operations.

Create a queue object

1. import queue
3. q = queue.Queue(maxsize = 10)

The queue.Queue class is a synchronous implementation of a queue. The queue length can be unlimited or limited. The queue length can be set through the optional parameter maxsize of the Queue constructor. If maxsize is less than 1, the queue length is unlimited.

Put a value into the queue

1. q.put("a")

Call the put() method of the queue object to insert an item at the end of the queue. put() has two parameters. The first item is required and is the value of the inserted item; the second block is an optional parameter and defaults to 1. If the queue is currently empty and block is 1, the put() method causes the calling thread to pause until a data unit is freed. If block is 0, the put() method will throw a Full exception.

Remove a value from the queue

1. q.get()

Call the get() method of the queue object to delete and return an item from the head of the queue. The optional parameter is block, which defaults to True. If the queue is empty and block is True, get() causes the calling thread to pause until an item becomes available. If the queue is empty and block is False, the queue will throw an Empty exception.

Python Queue module has three queues and constructors

1. # First in first out
2. class queue.Queue(maxsize=0)
3. First in, last out
4. class queue.LifoQueue(maxsize=0)
5. # The lower the priority queue level, the first it goes out
6. ##class queue.PriorityQueue(maxsize=0)
Common methods
1. q = queue.Queue()
2. # Returns the size of the queue
3. q.qsize()
4. # If the queue is empty, return True, otherwise False
5. q.empty()
6. If the queue is full, return True, otherwise False
7. q.full()
8. Get the queue, timeout waiting time
9. q.get([block[,timeout]])
10. # Equivalent to q.get(False)
11. ##q.get_nowait()
12. # Wait until the queue is empty before performing other operations
13. q.join()

Producer-Consumer Model

Using the producer and consumer patterns in development and programming can solve most concurrency problems. This mode improves the overall data processing speed of the program by balancing the work capabilities of the production thread and the consumer thread.

Why use the producer and consumer model

In the thread world, the producer is the thread that produces data, and the consumer is the thread that consumes data. In multi-threaded development, if the producer's processing speed is very fast and the consumer's processing speed is very slow, then the producer must wait for the consumer to finish processing before continuing to produce data. In the same way, if the processing power of the consumer is greater than the producer, then the consumer must wait for the producer. In order to solve this problem, the producer and consumer models were introduced.

What is the producer-consumer model

The producer-consumer model uses a container to solve the strong coupling problem between producers and consumers. Producers and consumers do not communicate directly with each other, but communicate through blocking queues. Therefore, after the producer produces the data, it no longer waits for the consumer to process it, but directly throws it to the blocking queue. The consumer does not ask the producer for data, but Take it directly from the blocking queue. The blocking queue is equivalent to a buffer, balancing the processing capabilities of producers and consumers.

The most basic example of the producer-consumer model.

1. import queue,threading,time
3. # #q = queue.Queue(maxsize=10)
5. ##def Producer():
6. count = 1
7. ##while
   True:
        q.put("
8. bone%s
   "%count)
                                                                                                              "
9. produced bones
   ",count)
Count += 1
def Consumer(name):
12. ##​
while
13. q.qsize () > 0:
    ##                  print("
    [%s] got [%s] and ate it... "%(name,q.
14. get
    ()))## time.sleep(1)
##p = threading.Thread(target=Producer,)
17. c1 = threading.Thread(target=Consumer,args=("王财",))
18. ##c2 = threading.Thread (target=Consumer,args=("来福",))
19. ##p.start()
c1.start()
20. ##c2.start()