The correct way to open multi-process shared variables based on python

The following article will share with you the correct way to open multi-process shared variables based on python. It has a good reference value and I hope it will be helpful to everyone. Let’s take a look together

Multiple processes share variables and obtain results

Due to engineering requirements, multiple threads must be used to run a program. But because I heard that python’s multi-threading is fake, I used multi-process. Anyway, the tasks need to share fewer parameters.

After consulting the information, I found that Multiprocessing is mainly used to implement multi-process. There are two ways, one is Process and the other is Pool.

p = Process(target=fun,args=(args))

Then use p.start() to start a child process, and use the p.join() method to make the child process run. Execute the parent process after completion.

But this is very annoying, and I have to write a for loop to open n threads and join.

So it is recommended to use Pool. It can open a fixed-size process pool, and then each thread executes the apply_async() function to call the function to be executed, and finally closes and joins.

The code is as follows:

##

pathm=Manager().Queue(len(pathlist))
for d in pathlist:
 pathm.put(d)
p=Pool(cp.threads)
results=[]
for i in range(cp.threads):
 temp=p.apply_async(ProcessWorker,args=(i,pathm,cp))
 results.append(temp)
print 'Waiting for all subprocesses done...'
p.close()
p.join()
print 'All subprocesses finish Processing.'
results=[r.get() for r in results]

The above code demonstration Learn how to use pool multi-process, how to share the variable pathm between processes in the Pool, and how to obtain the results of process function execution. It should be noted that ProcessWorker must be an unbounded function, otherwise an error will be reported and the function cannot be pickled and cannot be assigned to each process.

cPickle.PicklingError: Can&#39;t pickle <type &#39;instancemethod&#39;>: attribute lookup __builtin__.instancemethod failed

Bounded functions and python’s multi-process mechanism

A concept derived from the above is the concept of bounded function and unbounded function.

After reviewing the information, I concluded as follows:

Bounded functions are packaged in a class, and can only be used when the class is instantiated The function used is bounded by this instance. We often call these functions class methods. For example, a class method that takes self as a parameter.

Unbounded functions can be functions that are not wrapped in a class, or they can be static methods in a class. They are independent of the class. For example, a static method in a class cannot access parameters and other methods in the class even if it is defined in a class.

Python's multi-process mechanism should compile and package the methods to be called by each process and the parameters passed in (such as ProcessWorker in the above example), and then copy them to each process for execution. If the input is a bounded function, then its parameters should be the class to which it belongs (including parameters and methods), but this cannot be obtained, and class attributes and methods may have pitfalls, making it difficult to package. Therefore, Python restricts that the functions to be called by multiple processes cannot be class methods.

We need to put the functions called by multiple processes outside the class, or turn them into static functions. However, static functions cannot be called by methods of the class to which they belong (in the form of self.ProcessWorker). They need to be called externally, such as mc=MyClass(), mc.ProcessWorker, or MyClass().ProcessWorker.

Related recommendations:

Methods in decorated classes based on Python decorators

Arrays and matrices based on Python Numpy Detailed explanation of matrix_python

The above is the detailed content of The correct way to open multi-process shared variables based on python. For more information, please follow other related articles on the PHP Chinese website!