Efficiency comparison experiment of single thread, multi-thread and multi-process in Python

Comparative experiment

The data shows that if the multi-threaded process is CPU-intensive, multi-threading will not improve the efficiency much. On the contrary, the efficiency may decrease due to frequent switching of threads. It is recommended to use multi-threading; If it is IO-intensive, the multi-threaded process can use the idle time while waiting for IO blocking to execute other threads to improve efficiency. So we compare the efficiency of different scenarios based on experiments

(1) Introduce the required modules

import requests
import time
from threading import Thread
from multiprocessing import Process

(2) Define CPU-intensive computing functions

def count(x, y):
    # 使程序完成150万计算
    c = 0
    while c < 500000:
        c += 1
        x += x
        y += y

(3) Define IO-intensive file reading and writing functions

def write():
    f = open("test.txt", "w")
    for x in range(5000000):
        f.write("testwrite\n")
    f.close()

def read():
    f = open("test.txt", "r")
    lines = f.readlines()
    f.close()

(4) Define the network request function

_head = {
            'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36'}
url = "http://www.tieba.com"
def http_request():
    try:
        webPage = requests.get(url, headers=_head)
        html = webPage.text
        return {"context": html}
    except Exception as e:
        return {"error": e}

(5) Test the time required for linear execution of IO-intensive operations, CPU-intensive operations, and time required for network request-intensive operations

# CPU密集操作
t = time.time()
for x in range(10):
    count(1, 1)
print("Line cpu", time.time() - t)

# IO密集操作
t = time.time()
for x in range(10):
    write()
    read()
print("Line IO", time.time() - t)

# 网络请求密集型操作
t = time.time()
for x in range(10):
    http_request()
print("Line Http Request", time.time() - t)

Output

CPU-intensive: 95.6059999466, 91.57099986076355 92.52800011634827, 99.96799993515015

IO intensive: 24.25, 21.76699995994568, 21.769999980926514, 22.060999870300293

Network request intensive: 4.519999980926514, 8 .563999891281128, 4.371000051498413, 4.522000074386597, 14.671000003814697

(6) Test the time required for multi-threaded concurrent execution of CPU-intensive operations

counts = []
t = time.time()
for x in range(10):
    thread = Thread(target=count, args=(1,1))
    counts.append(thread)
    thread.start()

e = counts.__len__()
while True:
    for th in counts:
        if not th.is_alive():
            e -= 1
    if e <= 0:
        break
print(time.time() - t)

Output: 99.9240000248, 101.26400017738342, 102.32200002670288

(7) Test the time required for multi-threaded concurrent execution of IO-intensive operations

def io():
    write()
    read()

t = time.time()
ios = []
t = time.time()
for x in range(10):
    thread = Thread(target=count, args=(1,1))
    ios.append(thread)
    thread.start()

e = ios.__len__()
while True:
    for th in ios:
        if not th.is_alive():
            e -= 1
    if e <= 0:
        break
print(time.time() - t)

Output: 25.69700002670288, 24.024000167846 68

(8) Test the time required for multi-threaded concurrent execution of network-intensive operations

t = time.time()
ios = []
t = time.time()
for x in range(10):
    thread = Thread(target=http_request)
    ios.append(thread)
    thread.start()

e = ios.__len__()
while True:
    for th in ios:
        if not th.is_alive():
            e -= 1
    if e <= 0:
        break
print("Thread Http Request", time.time() - t)

Output: 0.7419998645782471, 0.3839998245239258, 0.3900001049041748

(9) Test the time required for multiple processes to concurrently perform CPU-intensive operations

counts = []
t = time.time()
for x in range(10):
    process = Process(target=count, args=(1,1))
    counts.append(process)
    process.start()
e = counts.__len__()
while True:
    for th in counts:
        if not th.is_alive():
            e -= 1
    if e <= 0:
        break
print("Multiprocess cpu", time.time() - t)

Output: 54.342000007629395, 53.437999 963760376

(10) Test the concurrent execution of IO-intensive operations by multiple processes

t = time.time()
ios = []
t = time.time()
for x in range(10):
    process = Process(target=io)
    ios.append(process)
    process.start()

e = ios.__len__()
while True:
    for th in ios:
        if not th.is_alive():
            e -= 1
    if e <= 0:
        break
print("Multiprocess IO", time.time() - t)

Output: 12.509000062942505, 13.059000015258789

(11) Test multi-process concurrent execution of Http request-intensive operations

t = time.time()
httprs = []
t = time.time()
for x in range(10):
    process = Process(target=http_request)
    ios.append(process)
    process.start()

e = httprs.__len__()
while True:
    for th in httprs:
        if not th.is_alive():
            e -= 1
    if e <= 0:
        break
print("Multiprocess Http Request", time.time() - t)

Output: 0.5329999923706055, 0.4760000705718994

Experimental results

Through the above results, we can see:

Multiple threads are IO intensive There doesn't seem to be a big advantage under the type of operation (perhaps the advantage will be reflected if the IO operation task is heavier). Under the CPU-intensive operation, the performance is obviously worse than the single-threaded linear execution, but for network requests, For operations that block threads such as busy waiting, the advantages of multi-threading are very obvious

Multiple processes can show performance advantages in CPU-intensive, IO-intensive, and network request-intensive operations (thread-blocking operations often occur). However, for network request-intensive operations, it is almost the same as multi-threading, but it takes up more resources such as CPU, so in this case, we can choose multi-threading to execute