How to Share a Queue Safely Between Worker Processes in Python\'s Multiprocessing Module?

Sharing a Queue among Multiple Worker Processes

The multiprocessing module provides mechanisms for creating and managing multiple processes for performing tasks concurrently. One common use case is to have worker processes report their results back to a central process. This is where queues come into play. However, when using apply_async to spawn worker processes, sharing a queue between them can be challenging.

The Error: Queue Objects for Inheritance Only

When attempting to pass a queue directly to apply_async, you may encounter a RuntimeError: "Queue objects should only be shared between processes through inheritance." This error signifies that queue instances can only be shared between processes that are directly related by inheritance. In the provided code, the worker process is not inherited from the main process, hence the error.

Solution: Using multiprocessing.Manager

To overcome this inheritance restriction, multiprocessing provides the Manager class. Manager provides a way to create objects that can be shared across multiple processes, including queues. By utilizing Manager, we can create a queue that is accessible to all worker processes without the need for inheritance.

Refactored Code:

The following code demonstrates how to share a queue among worker processes using multiprocessing.Manager:

import multiprocessing
def worker(name, que):
    que.put("%d is done" % name)

if __name__ == '__main__':
    pool = multiprocessing.Pool(processes=3)
    m = multiprocessing.Manager()
    q = m.Queue()
    workers = pool.apply_async(worker, (33, q))

In this code, Manager() creates a manager instance, m, which is then used to instantiate a shared queue, q. The queue is passed as an argument to the apply_async method, allowing the worker processes to access and write results to it. This approach enables communication between worker processes and the main process through the shared queue, eliminating the potential for inheritance errors.

The above is the detailed content of How to Share a Queue Safely Between Worker Processes in Python\'s Multiprocessing Module?. For more information, please follow other related articles on the PHP Chinese website!