Home > Article > Backend Development > How to effectively manage and control the concurrency of parallel bash subprocesses in Python?
Parallel Bash Subprocesses with Python: A Comprehensive Guide
Utilizing the Python threading and subprocess modules effectively can help you execute multiple bash processes simultaneously. However, simply creating threads with threading may not achieve the desired parallelism.
Concurrent Process Management Without Threads
A straightforward approach to running bash processes concurrently is to avoid using threads altogether. Using the subprocess.Popen utility, you can directly invoke multiple commands in parallel, as demonstrated below:
<code class="python">from subprocess import Popen commands = [ 'date; ls -l; sleep 1; date', 'date; sleep 5; date', 'date; df -h; sleep 3; date', 'date; hostname; sleep 2; date', 'date; uname -a; date', ] # Execute commands concurrently processes = [Popen(cmd, shell=True) for cmd in commands]</code>
Controlling Concurrency with Multiprocessing
If you need to limit the number of concurrent processes, you can employ the multiprocessing.dummy.Pool, which provides a thread-based interface similar to multiprocessing.Pool. The following code illustrates this approach:
<code class="python">from functools import partial from multiprocessing.dummy import Pool from subprocess import call pool = Pool(2) # Limit to 2 concurrent processes for i, returncode in enumerate(pool.imap(partial(call, shell=True), commands)): if returncode != 0: print("%d command failed: %d" % (i, returncode))</code>
Non-Blocking Child Process Management
Alternatively, you can limit concurrent child processes without resorting to thread or process pools. The code below demonstrates this strategy:
<code class="python">from subprocess import Popen from itertools import islice max_workers = 2 # Maximum number of concurrent processes processes = (Popen(cmd, shell=True) for cmd in commands) running_processes = list(islice(processes, max_workers)) # Start initial processes while running_processes: for i, process in enumerate(running_processes): if process.poll() is not None: # Process has completed running_processes[i] = next(processes, None) # Start new process if running_processes[i] is None: # No new processes del running_processes[i] break</code>
For Unix systems, consider using os.waitpid(-1, 0) to avoid busy loops and wait for any child process to terminate.
The above is the detailed content of How to effectively manage and control the concurrency of parallel bash subprocesses in Python?. For more information, please follow other related articles on the PHP Chinese website!