Python multi-process usage example (code)

The content of this article is about the usage examples (code) of python multi-process. It has certain reference value. Friends in need can refer to it. I hope it will be helpful to you.

Python multi-threading is suitable for IO-intensive scenarios, but in CPU-intensive scenarios, multi-core CPUs cannot be fully utilized. Coroutines are essentially thread-based and cannot fully utilize the advantages of multi-cores.

For computing-intensive scenarios that require the use of multiple processes, python's multiprocessing is very similar to the threading module and supports batch creation of child processes using a process pool.

• Create a single Process process (using func)

You only need to instantiate the Process class and pass the function to the target parameter. This is the same as the threading module Very similar, args is the parameter of the function

import os
from multiprocessing import Process

# 子进程要执行的代码
def task(name):
    print('run child process %s (%s)...' % (name, os.getpid()))

if __name__ == '__main__':
    print('parent process %s.' % os.getpid())
    p = Process(target=task, args=('test',))
    p.start()
    p.join()
    print('process end.')

• ##Create a single Process process (using class)

Inherit the Process class and override the run method to create a process, which is basically the same as the threading module

import multiprocessing
import os
from multiprocessing import current_process
class Worker(multiprocessing.Process):
    def run(self):
        name = current_process().name  # 获取当前进程的名称
        print(&#39;run child process <%s>  (%s)&#39; % (name, os.getpid()))

        print(&#39;In %s&#39; % self.name)
        return

if __name__ == &#39;__main__&#39;:
    print(&#39;parent process %s.&#39; % os.getpid())
    p = Worker()
    p.start()
    p.join()
    print(&#39;process end.&#39;)

　* Stop the process

terminate() ends the child process, but the resources of the child process cannot be released. This is not recommended because the running status of the child thread is not clear when it ends, and there is a high possibility that the child thread will be terminated at an inappropriate moment. Finish.

import multiprocessing
import time

def worker():
    print('starting worker')
    time.sleep(0.1)
    print('finished worker')

if __name__ == '__main__':
    p = multiprocessing.Process(target=worker)
    print('执行前:', p.is_alive())
    p.start()
    print('执行中:', p.is_alive())
    p.terminate()  # 发送停止号
    print('停止:', p.is_alive())
    p.join()
    print('等待完成:', p.is_alive())

• Create multiple Process processes directly

import multiprocessing

def worker(num):
    print(f'Worker:%s %s', num)
    return

if __name__ == '__main__':
    jobs = []
    for i in range(5):
        p = multiprocessing.Process(target=worker, args=(i,))
        jobs.append(p)
        p.start()

• Use process pool to create multiple processes

When using Python for system management, especially when operating multiple file directories at the same time, or Remote control of multiple hosts and parallel operations can save a lot of time. When the number of objects to be operated is not large, you can directly use Process in multiprocessing to dynamically generate multiple processes. A dozen or so are fine, but if there are hundreds or thousands of targets, it would be too much to manually limit the number of processes. If it is too cumbersome, the process pool can be used at this time.

Pool can provide a specified number of processes for users to call. When a new request is submitted to the pool, if the pool is not full, a new process will be created to execute the request; but if the pool If the number of processes has reached the specified maximum, the request will wait until a process in the pool ends, and then a new process will be created to handle it.

import os
import random
import time
from multiprocessing import Pool
from time import ctime


def task(name):
    print('start task %s (%s)...' % (name, os.getpid()))
    start = time.time()
    time.sleep(random.random() * 3)

    print('end task %s runs %0.2f seconds.' % (name, (time.time() - start)))


if __name__ == '__main__':
    print('parent process %s.' % os.getpid())

    p = Pool()  # 初始化进程池
    for i in range(5):
        p.apply_async(task, args=(i,))  # 追加任务 apply_async 是异步非阻塞的，就是不用等待当前进程执行完毕，随时根据系统调度来进行进程切换。

    p.close()

    p.join()  # 等待所有结果执行完毕，会等待所有子进程执行完毕，调用join()之前必须先调用close()
    print(f'all done at: {ctime()}')

If you care about the execution result of each process, you can use the get method to return the result. The code is as follows

import os
import random
import time
from multiprocessing import Pool, current_process
from time import ctime


def task(name):
    print('start task %s (%s)...' % (name, os.getpid()))
    start = time.time()
    time.sleep(random.random() * 3)
    print('end task %s runs %0.2f seconds.' % (name, (time.time() - start)))

    return current_process().name + 'done'

if __name__ == '__main__':
    print('parent process %s.' % os.getpid())

    result = []

    p = Pool()  # 初始化进程池
    for i in range(5):
        result.append(p.apply_async(task, args=(i,)))  # 追加任务 apply_async 是异步非阻塞的，就是不用等待当前进程执行完毕，随时根据系统调度来进行进程切换。

    p.close()

    p.join()  # 等待所有结果执行完毕

    for res in result:
        print(res.get())  # get()函数得出每个返回结果的值

    print(f'all done at: {ctime()}')

　

The above is the detailed content of Python multi-process usage example (code). For more information, please follow other related articles on the PHP Chinese website!