How to Handle Keyboard Interrupts in Python\'s Multiprocessing Pool?

Handling Keyboard Interrupts in Python's Multiprocessing Pool

In Python's multiprocessing module, the Pool class provides a convenient way to distribute tasks across multiple processes. However, handling KeyboardInterrupt events in Pools can be challenging, as demonstrated by the code snippet:

<code class="python">from multiprocessing import Pool
from time import sleep
from sys import exit

def slowly_square(i):
    sleep(1)
    return i*i

def go():
    pool = Pool(8)
    try:
        results = pool.map(slowly_square, range(40))
    except KeyboardInterrupt:
        # **** THIS PART NEVER EXECUTES. ****
        pool.terminate()
        print "You cancelled the program!"
        sys.exit(1)
    print "\nFinally, here are the results: ", results

if __name__ == "__main__":
    go()</code>

When running this code, pressing Ctrl C will not trigger the cleanup process, leaving the subprocesses running indefinitely. To address this issue, consider the following workaround:

The behavior observed in the code is a consequence of a Python bug. KeyboardInterrupt is not sent when waiting for a condition in threading.Condition.wait(). As Pool.map() uses a condition wait internally, the interrupt is never received.

A solution is to use Pool.map_async(), which allows specifying a timeout. By setting a sufficiently long timeout (e.g., 9999999), we can ensure that the interrupt will be triggered within a reasonable time.

Therefore, replace:

<code class="python">    results = pool.map(slowly_square, range(40))</code>

with:

<code class="python">    results = pool.map_async(slowly_square, range(40)).get(9999999)</code>

This workaround provides a way to gracefully handle KeyboardInterrupt events in multiprocessing Pools, allowing for the termination of all subprocesses when the user cancels the program.

The above is the detailed content of How to Handle Keyboard Interrupts in Python\'s Multiprocessing Pool?. For more information, please follow other related articles on the PHP Chinese website!