How to Stream Subprocess Output Line by Line in Python?

Streaming Subprocess Output Line by Line

When dealing with noisy Linux utilities invoked via subprocess in Python, efficiently capturing output both to a log file and displaying it to the user can be challenging.

The Issue:

Initially, a Python script attempted to capture subprocess output using the following code:

for line in proc.stdout:
    # Filter output here
    print("test:", line.rstrip())

However, this approach did not exhibit the desired behavior of streaming output line by line. Instead, output only appeared after a significant amount had been generated.

The Solution:

The issue lies in the use of the iterator for line in proc.stdout, which reads the entire output upfront. To address this, readline() should be used instead:

# Filter output
import subprocess

proc = subprocess.Popen(['python', 'fake_utility.py'], stdout=subprocess.PIPE)
while True:
    line = proc.stdout.readline()
    if not line:
        break
    # Filter output here
    print("test:", line.rstrip())

With this modification, output is now streamed line by line as desired.

Buffering Considerations:

It's important to note that the subprocess may still buffer output. To handle this, the flush method or other techniques may be necessary to ensure timely display of output.

The above is the detailed content of How to Stream Subprocess Output Line by Line in Python?. For more information, please follow other related articles on the PHP Chinese website!