How to Gracefully Handle KeyboardInterrupt with Multiprocessing Pools in Python

Keyboard Interruption Handling in Multiprocessing Pools: A Python Pitfall

In Python's multiprocessing module, KeyboardInterrupt events seemingly fail to terminate worker processes within a Pool. Consider the code snippet:

<code class="python">from multiprocessing import Pool
from time import sleep
from sys import exit

def slowly_square(i):
    sleep(1)
    return i*i

def go():
    pool = Pool(8)
    try:
        results = pool.map(slowly_square, range(40))
    except KeyboardInterrupt:
        # **** THIS PART NEVER EXECUTES. ****
        pool.terminate()
        print "You cancelled the program!"
        sys.exit(1)
    print "\nFinally, here are the results: ", results

if __name__ == "__main__":
    go()</code>

When you press Ctrl C to raise a KeyboardInterrupt, the code hangs indefinitely instead of gracefully terminating the Pool. This is due to a Python bug that prevents KeyboardInterrupt from interrupting calls to threading.Condition.wait().

Workaround:

A workaround solution is to specify a timeout for Pool operations. Replace:

<code class="python">    results = pool.map(slowly_square, range(40))</code>

with:

<code class="python">    results = pool.map_async(slowly_square, range(40)).get(9999999)</code>

By specifying an arbitrarily large timeout, you effectively remove the blocking behavior and allow the interrupt to be processed promptly. This will cause all worker processes to exit gracefully when Ctrl C is pressed.

The above is the detailed content of How to Gracefully Handle KeyboardInterrupt with Multiprocessing Pools in Python. For more information, please follow other related articles on the PHP Chinese website!