


This article brings you a detailed introduction (code example) about multi-threading in Python. It has certain reference value. Friends in need can refer to it. I hope it will be helpful to you.
This article records the problems encountered in learning Python and some common usages. Note that the Python version of this development environment is 2.7.
1. Python file naming
When naming python files, you must be careful not to conflict with the system's default module name, otherwise an error will be reported.
As shown in the following example, when learning threads, the file name is threading.py
. The Python script is completely normal and there is no problem. As a result, the following error is reported: AttributeError: 'module' object has no attribute 'xxx'
.
threading.py
# -*- coding:utf-8 -*- """ @author: Corwien @file: threading_test.py @time: 18/8/25 09:14 """ import threading # 获取已激活的线程数 print(threading.active_count())
Execution:
➜ baseLearn python threading/threading.py Traceback (most recent call last): File "threading/threading.py", line 9, in <module> import threading File "/Users/kaiyiwang/Code/python/baseLearn/threading/threading.py", line 12, in <module> print(threading.active_count()) AttributeError: 'module' object has no attribute 'active_count' ➜ baseLearn</module></module>
Problem location:
View import
library Source file, found that the source file exists and has no errors, and the .pyc
file of the source file also exists
Problem solution:
1.When naming the py script, do not use the same words as python reserved words, module names, etc.
2. Delete the
.pyc
file of the library (because py The script will generate a .pyc file every time it is run; if the .pyc file has been generated, if the code is not updated, pyc will still be used when running, so delete the .pyc file) and rerun the code; or find a program that can In the environment where the code is running, copy and replace the .pyc file of the current machine.
Rename the script file name to threading_test.py
, and then execute it, it will not An error was reported.
➜ baseLearn python threading/threading_test.py 1 ➜ baseLearn
2. Multi-threading threading
Multi-threading is an effective way to speed up program calculations. Python’s multi-threading modulethreading
is quick and easy to get started. Let’s start with this section. How everyone uses it.
1. Add thread
threading_test.py
# -*- coding:utf-8 -*- """ @author: Corwien @file: threading_test.py @time: 18/8/25 09:14 """ import threading # 获取已激活的线程数 # print(threading.active_count()) # 查看所有线程信息 # print(threading.enumerate()) # 查看现在正在运行的线程 # print(threading.current_thread()) def thread_job(): print('This is a thread of %s' % threading.current_thread()) def main(): thread = threading.Thread(target=thread_job,) # 定义线程 thread.start() # 让线程开始工作 if __name__ == '__main__': main()
2. Join function
The result of not adding join()
We letT1
increase the time consuming of thread work
threading_join.py
# -*- coding:utf-8 -*- """ @author: Corwien @file: threading_join.py @time: 18/8/25 09:14 """ import threading import time def thread_job(): print('T1 start\n') for i in range(10): time.sleep(0.1) # 任务时间0.1s print("T1 finish\n") def main(): added_thread = threading.Thread(target=thread_job, name='T1') # 定义线程 added_thread.start() # 让线程开始工作 print("all done\n") if __name__ == '__main__': main()
The expected output results are in order Execute in sequence:
T1 start T1 finish all done
But the actual running result is:
➜ baseLearn python threading/threading_join.py T1 start all done T1 finish ➜ baseLearn
The result of joining()
The thread task is output before it is completedall done
. If you want to follow the order, you can call join
on the thread after starting it:
added_thread.start() added_thread.join() print("all done\n")
Print results:
➜ baseLearn python threading/threading_join.py T1 start T1 finish all done
Full script file:
# -*- coding:utf-8 -*- """ @author: Corwien @file: threading_join.py @time: 18/8/25 09:14 """ import threading import time def thread_job(): print('T1 start\n') for i in range(10): time.sleep(0.1) # 任务时间0.1s print("T1 finish\n") def main(): added_thread = threading.Thread(target=thread_job, name='T1') # 定义线程 added_thread.start() # 让线程开始工作 added_thread.join() print("all done\n") if __name__ == '__main__': main()
small Test your skills
If you add two threads, what is the printed output?
# -*- coding:utf-8 -*- """ @author: Corwien @file: threading_join.py @time: 18/8/25 09:14 """ import threading import time def T1_job(): print('T1 start\n') for i in range(10): time.sleep(0.1) # 任务时间0.1s print("T1 finish\n") def T2_job(): print("T2 start\n") print("T2 finish\n") def main(): thread_1 = threading.Thread(target=T1_job, name='T1') # 定义线程 thread_2 = threading.Thread(target=T2_job, name='T2') # 定义线程 thread_1.start() # 开启T1 thread_2.start() # 开启T2 print("all done\n") if __name__ == '__main__': main()
The "one" result output is:
T1 start T2 start T2 finish all done T1 finish
Now neither T1 nor T2 has join
. Note that "one" is said here because of the appearance of all done It completely depends on the execution speed of the two threads. It is entirely possible that T2 finish appears after all done. We cannot tolerate this messy execution method, so we need to use join to control it.
Let’s try adding thread_1.join() after T1 starts and before T2 starts.
:
thread_1.start() thread_1.join() # notice the difference! thread_2.start() print("all done\n")
Print results:
T1 start T1 finish T2 start all done T2 finish
You can see , T2 will wait for T1 to finish before starting to run.
3. Queue to store process results
Implementation function
The code implements the function, transfers the data in the data list, uses four threads to process it, and saves the result in the Queue , after the thread is executed, obtain the stored result from the Queue
Define a Queue
in the multi-threaded function to save the return value, replaces return
, Define a multi-threaded list and initialize a multi-dimensional data list for processing:
threading_queue.py
# -*- coding:utf-8 -*- """ @author: Corwien @file: threading_queue.py @time: 18/8/25 09:14 """ import threading import time from queue import Queue def job(l, q): for i in range(len(l)): l[i] = l[i] ** 2 q.put(l) #多线程调用的函数不能用return返回值 def multithreading(): q = Queue() #q中存放返回值,代替return的返回值 threads = [] data = [[1,2,3],[3,4,5],[4,4,4],[5,5,5]] for i in range(4): #定义四个线程 t = threading.Thread(target=job, args=(data[i], q)) #Thread首字母要大写,被调用的job函数没有括号,只是一个索引,参数在后面 t.start() #开始线程 threads.append(t) #把每个线程append到线程列表中 for thread in threads: thread.join() results = [] for _ in range(4): results.append(q.get()) #q.get()按顺序从q中拿出一个值 print(results) if __name__ == '__main__': multithreading()
When executing the above script, an error occurred:
➜ baseLearn python threading/threading_queue.py Traceback (most recent call last): File "threading/threading_queue.py", line 11, in <module> from queue import Queue ImportError: No module named queue</module>
I checked the reason and it was caused by the python version:
Solution: No module named 'Queue'
On Python 2, the module is named Queue, on Python 3, it was renamed to follow PEP8 guidelines (all lowercase for module names), making it queue. The class remains Queue on all versions (following PEP8).
Typically, the way you'd write version portable imports would be to do:
In python3, we can quote like this:
try: import queue except ImportError: import Queue as queue
In python2, we can quote like this:
from Queue import Queue
Print:
baseLearn python ./threading/threading_queue.py [[1, 4, 9], [9, 16, 25], [16, 16, 16], [25, 25, 25]]
Full code:
threading_queue.py
# -*- coding:utf-8 -*- """ @author: Corwien @file: threading_queue.py @time: 18/8/25 09:14 """ import threading # import time from Queue import Queue def job(l, q): for i in range(len(l)): l[i] = l[i] ** 2 q.put(l) #多线程调用的函数不能用return返回值 def multithreading(): q = Queue() #q中存放返回值,代替return的返回值 threads = [] data = [[1,2,3],[3,4,5],[4,4,4],[5,5,5]] for i in range(4): #定义四个线程 t = threading.Thread(target=job, args=(data[i], q)) #Thread首字母要大写,被调用的job函数没有括号,只是一个索引,参数在后面 t.start() #开始线程 threads.append(t) #把每个线程append到线程列表中 for thread in threads: thread.join() results = [] for _ in range(4): results.append(q.get()) #q.get()按顺序从q中拿出一个值 print(results) if __name__ == '__main__': multithreading()
4. GIL efficiency issue
What is GIL?
This time we will take a look at why Python's multi-threading is sometimes not particularly ideal. The main reason is that there is a necessary link in the design of Python, which is Global Interpreter Lock (GIL)
. This thing allows Python to still process only one thing at a time.
Explanation of GIL:
尽管Python完全支持多线程编程, 但是解释器的C语言实现部分在完全并行执行时并不是线程安全的。 实际上,解释器被一个全局解释器锁保护着,它确保任何时候都只有一个Python线程执行。 GIL最大的问题就是Python的多线程程序并不能利用多核CPU的优势 (比如一个使用了多个线程的计算密集型程序只会在一个单CPU上面运行)。在讨论普通的GIL之前,有一点要强调的是GIL只会影响到那些严重依赖CPU的程序(比如计算型的)。 如果你的程序大部分只会涉及到I/O,比如网络交互,那么使用多线程就很合适, 因为它们大部分时间都在等待。实际上,你完全可以放心的创建几千个Python线程, 现代操作系统运行这么多线程没有任何压力,没啥可担心的。测试GIL
我们创建一个 job
, 分别用 threading 和 一般的方式执行这段程序. 并且创建一个 list 来存放我们要处理的数据. 在 Normal 的时候, 我们这个 list 扩展4倍, 在 threading 的时候, 我们建立4个线程, 并对运行时间进行对比.
threading_gil.py
# -*- coding:utf-8 -*- """ @author: Corwien @file: threading_gil.py @time: 18/8/25 09:14 """ import threading from Queue import Queue import copy import time def job(l, q): res = sum(l) q.put(l) #多线程调用的函数不能用return返回值 def multithreading(l): q = Queue() #q中存放返回值,代替return的返回值 threads = [] for i in range(4): #定义四个线程 t = threading.Thread(target=job, args=(copy.copy(l), q), name="T%i" % i) #Thread首字母要大写,被调用的job函数没有括号,只是一个索引,参数在后面 t.start() #开始线程 threads.append(t) #把每个线程append到线程列表中 [t.join() for t in threads] total = 0 for _ in range(4): total = q.get() #q.get()按顺序从q中拿出一个值 print(total) def normal(l): total = sum(l) print(total) if __name__ == '__main__': l = list(range(1000000)) s_t = time.time() normal(l*4) print('normal:', time.time() - s_t) s_t = time.time() multithreading(l) print('multithreading: ', time.time() - s_t)
如果你成功运行整套程序, 你大概会有这样的输出. 我们的运算结果没错, 所以程序 threading 和 Normal 运行了一样多次的运算. 但是我们发现 threading 却没有快多少, 按理来说, 我们预期会要快3-4倍, 因为有建立4个线程, 但是并没有. 这就是其中的 GIL 在作怪.
1999998000000 normal: 0.10034608840942383 1999998000000 multithreading: 0.08421492576599121
5、线程锁Lock
不使用 Lock 的情况
threading_lock.py
# -*- coding:utf-8 -*- """ @author: Corwien @file: threading_lock.py @time: 18/8/25 09:14 """ import threading # 全局变量A的值每次加1,循环10次,并打印 def job1(): global A for i in range(10): A+=1 print('job1',A) # 全局变量A的值每次加10,循环10次,并打印 def job2(): global A for i in range(10): A+=10 print('job2',A) # 定义两个线程,分别执行函数一和函数二 if __name__== '__main__': A=0 t1=threading.Thread(target=job1) t2=threading.Thread(target=job2) t1.start() t2.start() t1.join() t2.join()
打印输出数据:
➜ baseLearn python ./threading/threading_lock.py ('job1', ('job2'1) , (11)'job1' ('job2', 22) ('job2', 32) ('job2', 42) ('job2', 52) ('job2', 62) ('job2', 72) ('job2', 82) ('job2', 92) ('job2', 102) , 12) ('job1', 103) ('job1', 104) ('job1', 105) ('job1', 106) ('job1', 107) ('job1', 108) ('job1', 109) ('job1', 110)
可以看出,打印的结果非常混乱
使用 Lock 的情况
lock在不同线程使用同一共享内存
时,能够确保线程之间互不影响,使用lock的方法是, 在每个线程执行运算修改共享内存之前,执行lock.acquire()
将共享内存上锁, 确保当前线程执行时,内存不会被其他线程访问,执行运算完毕后,使用lock.release()
将锁打开, 保证其他的线程可以使用该共享内存。
函数一和函数二加锁
def job1(): global A,lock lock.acquire() for i in range(10): A+=1 print('job1',A) lock.release() def job2(): global A,lock lock.acquire() for i in range(10): A+=10 print('job2',A) lock.release()
主函数中定义一个Lock
if __name__== '__main__': lock=threading.Lock() A=0 t1=threading.Thread(target=job1) t2=threading.Thread(target=job2) t1.start() t2.start() t1.join() t2.join()
完整代码:
# -*- coding:utf-8 -*- """ @author: Corwien @file: threading_lock.py @time: 18/8/25 09:14 """ import threading def job1(): global A,lock lock.acquire() for i in range(10): A+=1 print('job1',A) lock.release() def job2(): global A,lock lock.acquire() for i in range(10): A+=10 print('job2',A) lock.release() if __name__== '__main__': lock = threading.Lock() A=0 t1=threading.Thread(target=job1) t2=threading.Thread(target=job2) t1.start() t2.start() t1.join() t2.join()
打印输出:
➜ baseLearn python ./threading/threading_lock.py ('job1', 1) ('job1', 2) ('job1', 3) ('job1', 4) ('job1', 5) ('job1', 6) ('job1', 7) ('job1', 8) ('job1', 9) ('job1', 10) ('job2', 20) ('job2', 30) ('job2', 40) ('job2', 50) ('job2', 60) ('job2', 70) ('job2', 80) ('job2', 90) ('job2', 100) ('job2', 110)
从打印结果来看,使用lock
后,一个一个线程执行完。使用lock
和不使用lock,最后打印输出的结果是不同的。
相关推荐:
The above is the detailed content of Detailed introduction to multithreading in python (code example). For more information, please follow other related articles on the PHP Chinese website!

Python is easier to learn and use, while C is more powerful but complex. 1. Python syntax is concise and suitable for beginners. Dynamic typing and automatic memory management make it easy to use, but may cause runtime errors. 2.C provides low-level control and advanced features, suitable for high-performance applications, but has a high learning threshold and requires manual memory and type safety management.

Python and C have significant differences in memory management and control. 1. Python uses automatic memory management, based on reference counting and garbage collection, simplifying the work of programmers. 2.C requires manual management of memory, providing more control but increasing complexity and error risk. Which language to choose should be based on project requirements and team technology stack.

Python's applications in scientific computing include data analysis, machine learning, numerical simulation and visualization. 1.Numpy provides efficient multi-dimensional arrays and mathematical functions. 2. SciPy extends Numpy functionality and provides optimization and linear algebra tools. 3. Pandas is used for data processing and analysis. 4.Matplotlib is used to generate various graphs and visual results.

Whether to choose Python or C depends on project requirements: 1) Python is suitable for rapid development, data science, and scripting because of its concise syntax and rich libraries; 2) C is suitable for scenarios that require high performance and underlying control, such as system programming and game development, because of its compilation and manual memory management.

Python is widely used in data science and machine learning, mainly relying on its simplicity and a powerful library ecosystem. 1) Pandas is used for data processing and analysis, 2) Numpy provides efficient numerical calculations, and 3) Scikit-learn is used for machine learning model construction and optimization, these libraries make Python an ideal tool for data science and machine learning.

Is it enough to learn Python for two hours a day? It depends on your goals and learning methods. 1) Develop a clear learning plan, 2) Select appropriate learning resources and methods, 3) Practice and review and consolidate hands-on practice and review and consolidate, and you can gradually master the basic knowledge and advanced functions of Python during this period.

Key applications of Python in web development include the use of Django and Flask frameworks, API development, data analysis and visualization, machine learning and AI, and performance optimization. 1. Django and Flask framework: Django is suitable for rapid development of complex applications, and Flask is suitable for small or highly customized projects. 2. API development: Use Flask or DjangoRESTFramework to build RESTfulAPI. 3. Data analysis and visualization: Use Python to process data and display it through the web interface. 4. Machine Learning and AI: Python is used to build intelligent web applications. 5. Performance optimization: optimized through asynchronous programming, caching and code

Python is better than C in development efficiency, but C is higher in execution performance. 1. Python's concise syntax and rich libraries improve development efficiency. 2.C's compilation-type characteristics and hardware control improve execution performance. When making a choice, you need to weigh the development speed and execution efficiency based on project needs.


Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Article

Hot Tools

WebStorm Mac version
Useful JavaScript development tools

EditPlus Chinese cracked version
Small size, syntax highlighting, does not support code prompt function

PhpStorm Mac version
The latest (2018.2.1) professional PHP integrated development tool

DVWA
Damn Vulnerable Web App (DVWA) is a PHP/MySQL web application that is very vulnerable. Its main goals are to be an aid for security professionals to test their skills and tools in a legal environment, to help web developers better understand the process of securing web applications, and to help teachers/students teach/learn in a classroom environment Web application security. The goal of DVWA is to practice some of the most common web vulnerabilities through a simple and straightforward interface, with varying degrees of difficulty. Please note that this software

SublimeText3 English version
Recommended: Win version, supports code prompts!