Detailed explanation of multi-process implementation in Python (with examples)

This article brings you a detailed explanation of multi-process implementation in Python (with examples). It has certain reference value. Friends in need can refer to it. I hope it will be helpful to you.

The fork function creates a child process

Basic use

The Linux operating system provides a fork function to create a child process . fork() is located in Python’s os module.
Just use the imported os module.

import os
os.fork()

Every time the fork() function is called, the corresponding parent process will generate a child process.
For example, the following code:

import os
os.fork()
os.fork()
os.fork()

After execution, 8 processes will be generated.

The return value of the fork() function

The return value of the fork() function for the child process is always 0, while the return value for the parent process is the child process pid (process number).

Example

#!/usr/bin/env python
import os
import time

rt = os.fork()

if rt == 0:
    print(f"The child process is {os.getpid()} . His father is {os.getppid()}")  # os.getpid()获取当前进程进程号，os.getppid()获取当前进程的父进程号
    time.sleep(5)
else:
    print(f"The father process is {os.getpid()} . His father is {os.getppid()}")
    time.sleep(5)

print(f"Now the process is {os.getpid()} . His father is {os.getppid()}")

Execution result:

##Process module

Import module

Python also provides the multiprocessing library to provide multi-threaded programming for the entire platform.

import multiprocessing

Simple process

The following code is a simple process:

from multiprocessing import Process


def work(num):
    for i in range(10):
        num += 1
    print(num)
    return 0


def main():
    num = 1
    p1 = Process(target = work, args = (num,))
    p1.start()


if __name__ == '__main__':
    main()

Here the Process class is introduced from the multiprocessing library.

p1 = Process(target = work, args = (num,)) is to create a process. The target is the function to perform the task, and args is the received parameters, which must be given in the form of tuples.
start() starts the process.
There are some methods for simultaneous processes:

join method

The join method of Process is similar to multi-threading. Waiting for the process to end.

Usage: join(timeout).
Using join(), the program will wait for the process to end before continuing with the following code.
If the timeout parameter is added, the program will wait for timeout seconds before continuing to execute the following program.

close method

close() is used to close the process, but it cannot close the running child process.

Process class

You can implement multiple processes by creating a class:

from multiprocessing import Process
import time


class My_Process(Process):

    def __init__(self,num):
        Process.__init__(self)
        self.num = num

    def run(self):
        time.sleep(2)
        print(self.num)


def main():
    for i in range(10):
        p = My_Process(i)
        p.start()


if __name__ == '__main__':
    main()

Process pool

from multiprocessing import Pool
import time


def target(num):
    time.sleep(2)
    print(num)


def main():
    pool = Pool(3)
    for i in range(3):
        pool.apply_async(target,(i,))
    pool.close()
    pool.join()
    print('Finish!!!')


if __name__ == '__main__':
    main()

Calling the join() method on the Pool object will wait for all child processes to complete execution. Close() must be called before calling join(). After calling close(), new processes cannot be added.

The num in Pool(num) is the number of processes to be added to it. If the number of processes is not specified, it defaults to the number of CPU cores.

Processes are independent of each other

Each process in multiple processes has a copy of the variables, and the operations between processes do not affect each other.

import multiprocessing
import time

zero = 0

def change_zero():
    global zero
    for i in range(3):
        zero = zero + 1
        print(multiprocessing.current_process().name, zero)

if __name__ == '__main__':
    p1 = multiprocessing.Process(target = change_zero)
    p2 = multiprocessing.Process(target = change_zero)
    p1.start()
    p2.start()
    p1.join()
    p2.join()
    print(zero)

The final execution result:

If file IO operations are performed, multiple processes will write to the same file.

Queue

Using Queue in multiprocessing allows different processes to access the same resources.

from multiprocessing import Process, Queue
def addone(q):
    q.put(1)
def addtwo(q):
    q.put(2)
if __name__ == '__main__':
    q = Queue()
    p1 = Process(target=addone, args = (q, ))
    p2 = Process(target=addtwo, args = (q, ))
    p1.start()
    p2.start()
    p1.join()
    p2.join()
    print(q.get())
    print(q.get())

The above is the detailed content of Detailed explanation of multi-process implementation in Python (with examples). For more information, please follow other related articles on the PHP Chinese website!