How to Process Gigabyte-Sized Files Line by Line Without Running Out of Memory?

How to Read a Massive File Line by Line Without Overloading Memory

Reading lengthy files line by line can be a challenge, especially when the file size exceeds your system's memory capacity. In such scenarios, attempting to load the entire file into memory using standard methods often results in out of memory errors.

Consider the following code snippet:

with open('inputfile.txt', 'r') as f:
    for line in f:
        # Process the line

When executing this code on a large file, the entire file is loaded into memory, which can cause problems if the file is in the gigabyte range.

To avoid this issue, consider using the fgets() function, which allows you to read the file line by line without loading the entire file into memory:

with open('inputfile.txt', 'r') as f:
    while True:
        line = fgets(f)
        if not line: break
        # Process the line

This approach reads the file line by line, freeing up memory once each line is processed, preventing out of memory errors.

The above is the detailed content of How to Process Gigabyte-Sized Files Line by Line Without Running Out of Memory?. For more information, please follow other related articles on the PHP Chinese website!