How Can I Stream Subprocess Output Line by Line in Python?

Streaming Output from subprocess.communicate()

Python's subprocess.communicate() function is useful for capturing the stdout of a process. However, it typically returns all the output at once. This can be inconvenient for processes that generate output over a prolonged period. To address this, we explore a technique for printing each line of the process's stdout in a streaming fashion.

Solution

By employing an iterative mechanism, we can retrieve lines of stdout as soon as they become available:

from subprocess import Popen, PIPE

# Launch a process with buffering set to 1 byte
p = Popen(["cmd", "arg1"], stdout=PIPE, bufsize=1)

# Iterate over the stdout lines as they become available
with p.stdout:
    for line in iter(p.stdout.readline, b''):
        print(line, end='')

# Wait for the process to terminate
p.wait()

Explanation

• iter(p.stdout.readline, b''): This creates an iterator that continuously reads lines from p.stdout in a non-blocking manner.
• end='': This ensures that each line is printed without a trailing newline character.
• p.wait(): Once the iterator is exhausted, this function waits for the process to finish before proceeding.

Additional Notes

• For Python 2, iter() is necessary to workaround a read-ahead bug.
• For Python 3, the universal_newlines option automatically decodes the stdout to text.
• Alternative methods, such as Pexpect or unbuffered utilities, can also be used to force unbuffered stdout.

The above is the detailed content of How Can I Stream Subprocess Output Line by Line in Python?. For more information, please follow other related articles on the PHP Chinese website!