Why Does Multiprocessing Fail with 'Can't Pickle InstanceMethod' and How Can I Fix It?

Error Handling in Multiprocessing: "Can't Pickle InstanceMethod"

In multiprocessing, the Pool.map() function divides tasks among processes for concurrent execution. However, you may encounter an error when using bound methods, as demonstrated in the given code:

import multiprocessing

def f(x):
    return x*x

def go():
    pool = multiprocessing.Pool(processes=4)        
    print pool.map(f, range(10))


if __name__== '__main__' :
    go()

This code works seamlessly. However, using a more object-oriented approach may fail with the error:

PicklingError: Can't pickle <type>: attribute lookup
__builtin__.instancemethod failed</type>

This error arises when you try to use Pool.map() with a bound method, as seen in the following code:

import someClass

if __name__== '__main__' :
    sc = someClass.someClass()
    sc.go()

class someClass(object):
    def __init__(self):
        pass

    def f(self, x):
        return x*x

    def go(self):
        pool = multiprocessing.Pool(processes=4)       
        print pool.map(self.f, range(10))

Resolution:

The issue stems from the requirement of multiprocessing to pickle objects to distribute them among processes. However, bound methods are not inherently picklable. The solution involves creating the necessary infrastructure to enable their pickling.

One approach is to use the copy_reg standard library method for registering pickling and unpickling functions. For instance, Steven Bethard's contribution to a related discussion thread offers a viable approach for method pickling and unpickling using copy_reg.

The above is the detailed content of Why Does Multiprocessing Fail with 'Can't Pickle InstanceMethod' and How Can I Fix It?. For more information, please follow other related articles on the PHP Chinese website!