Why Does My Python `subprocess.readline` Hang When Reading from a Continuously Writing C Program?

Python subprocess readline Hangs at "for line in iter"

Issue

When attempting to read the output of a C program, namely "2000," from a Python script using subprocess, the script hangs at the "for line in iter(process.stdout.readline, '')" line. This occurs despite the C program continuously printing "2000" and never reaching an end state.

Solution

The problem stems from a block buffering issue. By default, stdout in C programs running through pipes (e.g., when usingsubprocess.Popen) is block buffered. This means that data is not flushed to the pipe until the buffer is full or explicitly flushed.

Option 1: Modify C Program Directly

To resolve the issue, you can force stdout to be line buffered in the C program:

#include <stdio.h>

int main() {
    setvbuf(stdout, (char *) NULL, _IOLBF, 0); // Make line buffered stdout
    while (1) {
        printf("2000\n");
        sleep(1);
    }
    return 0;
}

Option 2: Use stdbuf Utility

Alternatively, you can use the stdbuf utility to change the buffering type without modifying the C program source:

from subprocess import Popen, PIPE

process = Popen(["stdbuf", "-oL", "./main"], stdout=PIPE, bufsize=1)
for line in iter(process.stdout.readline, b''):
    print(line)
process.communicate()

Option 3: Use Pseudo-TTY

Another approach involves using a pseudo-TTY to trick the C program into thinking it's running interactively:

import os
import pty
import sys
from select import select
from subprocess import Popen, STDOUT

master_fd, slave_fd = pty.openpty()
process = Popen("./main", stdin=slave_fd, stdout=slave_fd, stderr=STDOUT,
                bufsize=0, close_fds=True)
timeout = .1
with os.fdopen(master_fd, 'r+b', 0) as master:
    input_fds = [master, sys.stdin]
    while True:
        fds = select(input_fds, [], [], timeout)[0]
        # Handle subprocess output and user input here

Option 4: Use pexpect

Pexpect offers a higher-level interface that simplifies pseudo-TTY handling:

import pexpect

child = pexpect.spawn("./.main")
for line in child:
    print(line)
child.close()

The above is the detailed content of Why Does My Python `subprocess.readline` Hang When Reading from a Continuously Writing C Program?. For more information, please follow other related articles on the PHP Chinese website!