How to Resolve the \"Queue Objects Should Only Be Shared Between Processes Through Inheritance\" Error in Multiprocessing?

Managing Shared Queues with multiprocessing

In Python's multiprocessing module, creating a shared queue between processes requires careful consideration when employing multiprocessing.apply_async() to start asynchronous worker processes. The "Queue objects should only be shared between processes through inheritance." error occurs when attempting to pass a queue directly to workers.

To resolve this issue, utilize multiprocessing.Manager to manage the queue and make it accessible to worker processes. multiprocessing.Manager provides methods for creating manager objects that control and serialize various types of data in a shared memory segment.

The following code snippet demonstrates how to share a queue among multiple worker processes using multiprocessing.Manager:

<code class="python">import multiprocessing

def worker(name, que):
    que.put("%d is done" % name)

if __name__ == '__main__':
    pool = multiprocessing.Pool(processes=3)
    m = multiprocessing.Manager()
    q = m.Queue()
    workers = pool.apply_async(worker, (33, q))</code>

In this example, multiprocessing.Manager() creates a manager object (m) that manages shared data and provides proxy objects that can be passed to worker processes. By calling m.Queue(), we create a queue managed by the manager, which is accessible to all worker processes. The manager then serializes the proxy object and passes it to the worker, allowing it to communicate with the shared queue safely.

The above is the detailed content of How to Resolve the \"Queue Objects Should Only Be Shared Between Processes Through Inheritance\" Error in Multiprocessing?. For more information, please follow other related articles on the PHP Chinese website!