How Can I Monitor and Log Subprocess Output in Real-Time in Python?

Monitoring Live Output from Subprocess Commands

In Python, subprocess.Popen allows you to execute subprocesses while controlling their input and output. By default, Popen captures both stdout and stderr into a buffer that can be accessed later. However, this can be insufficient for real-time monitoring of the subprocess's progress.

Simultaneous Output Storing and Live Streaming

To simultaneously store the output for logging and provide a live stream, several approaches can be employed:

File-Based Approach

Create a file object for logging, pass it to Popen's stdout argument, and open the same file for reading in non-blocking mode. You can monitor the file's contents in a loop, writing them both to the console and the log file. This allows you to retrieve output incrementally.

with io.open("test.log", "wb") as writer, io.open("test.log", "rb", 1) as reader:
    process = subprocess.Popen(command, stdout=writer)
    while process.poll() is None:
        sys.stdout.write(reader.read())
        time.sleep(0.5)
    # Read the remaining
    sys.stdout.write(reader.read())

Iterator-Based Approach

Create an iterator from Popen's stdout and use it to continuously read and write to both the standard output and a log file. This approach is suitable when working with byte streams for both stdout and the log file.

import subprocess
import sys

with open("test.log", "wb") as f:
    process = subprocess.Popen(your_command, stdout=subprocess.PIPE)
    # replace "" with b"" for Python 3
    for c in iter(lambda: process.stdout.read(1), ""):
        sys.stdout.buffer.write(c)
        f.buffer.write(c)

Note: These approaches require explicit handling of binary data if necessary. For Python 3, replace "" with b"" and "w" with "wb" in the open statements.

The above is the detailed content of How Can I Monitor and Log Subprocess Output in Real-Time in Python?. For more information, please follow other related articles on the PHP Chinese website!