How to Achieve True Parallel Bash Process Execution in Python Without Relying on Threads?

How to Spawn Parallel Bash Processes with Python's threading and subprocess Modules?

Original Question:

How can I use Python's threading and subprocess modules to create parallel bash processes? Consecutive execution occurs instead of parallel execution when threads are initiated as described in this Stack Overflow response.

Answer:

Contrary to the original assumption, threads are not necessary for parallel bash subprocess execution. The following techniques provide various options:

Direct Execution (without Threads):

<code class="python">from subprocess import Popen

commands = ['date; ls -l; sleep 1; date', ...]
processes = [Popen(cmd, shell=True) for cmd in commands]</code>

Limited Concurrent Commands Using Threads with multiprocessing.dummy.Pool:

<code class="python">from functools import partial
from multiprocessing.dummy import Pool
from subprocess import call

pool = Pool(2)
for i, returncode in enumerate(pool.imap(partial(call, shell=True), commands)):
    ...</code>

Limit Concurrent Child Processes Without a Process Pool:

<code class="python">from subprocess import Popen
from itertools import islice

max_workers = 2
processes = (Popen(cmd, shell=True) for cmd in commands)
running_processes = list(islice(processes, max_workers))
while running_processes:
    ...</code>

Note for Unix Systems:

On Unix platforms, avoid the busy loop by blocking on os.waitpid(-1, 0) to wait for any child process to exit.

The above is the detailed content of How to Achieve True Parallel Bash Process Execution in Python Without Relying on Threads?. For more information, please follow other related articles on the PHP Chinese website!