Is it recommended to use multi-processing instead of multi-threading in Python? Share the reasons why it is recommended to use multi-process

Recently I’ve been reading about multi-threading in Python, and we often hear veterans say: “Multi-threading under Python is useless. , it is recommended to use multi-process!”, but why do you say this?​       
                                                                                                                                                                                                                                       So we have the following in-depth research: ​

##First of all, let’s emphasize the background:
1. What is GIL?
The full name of GIL is Global Interpreter Lock (global interpreter lock). The source is the consideration at the beginning of python design, for data security. decisions made.
2. Each CPU can only execute one thread at the same time
(In fact, multi-threading under a single-core CPU Both are just concurrency, not parallelism. From a macro perspective, concurrency and parallelism are both concepts of processing multiple requests at the same time, but there is a difference between concurrency and parallelism. Parallelism means that two or more events occur at the same time; concurrency means that two or more events occur at the same time. Two or more events occur within the same time interval.) Under Python multi-threading, the execution method of each thread: 1. Obtain GIL
2 , execute the code until sleep or the python virtual machine suspends it.
3. Release the GIL



##It can be seen that if a thread wants to execute, it must first get the GIL. We can think of the GIL as a "pass", and in a python process, there is only one GIL. Threads that cannot obtain a pass are not allowed to enter the CPU for execution.       
                                                                                                    
##In Python2.x , the release logic of GIL is that the current thread encounters an IO operation or the ticks count reaches 100 (ticks can be regarded as a counter of Python itself, specially used for GIL, and is reset to zero after each release. This count can be adjusted through sys.setcheckinterval ) to release.​
                                                                     Competing for locks and switching threads will consume resources. And because of the GIL lock, a process in Python can only execute one thread at the same time (the thread that has obtained the GIL can execute). This is why Python's multi-threading efficiency is not high on multi-core CPUs.                                                                                                Is multithreading completely useless?       
Here we have a classified discussion:                                                                                ​ etc.), in this case, due to the heavy calculation work, the ticks count will soon reach the threshold, and then trigger the release and re-competition of GIL (switching back and forth between multiple threads certainly consumes resources), so Multi-threading under python is not friendly to CPU-intensive code.

2. IO-intensive code (File processing, web crawlers, etc.), multi-threading can effectively improve efficiency (there will be IO waiting for IO operations under a single thread, causing unnecessary waste of time, and turning on multi-threading can automatically Switching to thread B will not waste CPU resources, thus improving program execution efficiency).
So python's multi-threading is more friendly to IO-intensive code.                                                                              , GIL does not use tick counting, but uses a timer (after the execution time reaches the threshold, the current thread releases the GIL), which is more friendly to CPU-intensive programs, but still does not solve the problem of the same time caused by GIL The problem is that only one thread can be executed, so the efficiency is still unsatisfactory.


Please note : Multi-core multi-threading is worse than single-core multi-threading. The reason is that under single-core multi-threading, every time the GIL is released, the thread that wakes up can obtain the GIL lock, so it can execute seamlessly. However, under multi-core, after CPU0 releases the GIL , the threads on other CPUs will compete, but the GIL may be obtained by CPU0 immediately, causing the awakened threads on several other CPUs to wait awake until the switching time and then enter the pending state. This will cause the threads to be Thrashing, resulting in lower efficiency##                                     #Back to the original question: We often hear veterans say:
"If you want to make full use of multi-core CPUs in Python, use multiple processes", what is the reason? Woolen cloth?​
##The reason is: each process has its own independent GIL , do not interfere with each other, so that they can be executed in parallel in a true sense, so in python, the execution efficiency of multi-process is better than that of multi-thread (only for multi-core CPU).                                                 So here is the conclusion: under multi-core, if you want to improve efficiency in parallel, a more common method is to use multi-process, which can effectively improve execution efficiency

[Related recommendations]

1. Examples of multi-process and multi-threading in Python (1)

2. Multi-process and multi-thread examples in Python (2) Programming methods

3. Is multi-process or multi-thread faster in Python?

4. Detailed introduction to Python processes, threads, and coroutines

5. Python concurrent programming thread pool/process pool

The above is the detailed content of Is it recommended to use multi-processing instead of multi-threading in Python? Share the reasons why it is recommended to use multi-process. For more information, please follow other related articles on the PHP Chinese website!