How Can I Efficiently Retrieve the Last 10 Lines of a Gigantic Text File?

Efficiently Retrieving the Last 10 Lines of a Massive Text File

In the realm of large-scale text processing, retrieving the final lines of an exceptionally large file poses a unique challenge. One such file, exceeding 10GB in size, presents a significant hurdle in efficiently obtaining this data.

To tackle this issue, an effective approach involves traversing the file backward, beginning from the end. We aim to locate ten consecutive newlines, indicating the presence of the desired lines. Subsequently, we read forward to capture these lines, considering potential encoding variations.

For instance, in C#, a comprehensive implementation handles cases where the file contains fewer than ten lines. The following code snippet exemplifies this approach:

public static string ReadEndLines(string path, Int64 numberOfLines, Encoding encoding, string lineSeparator) {

    int sizeOfChar = encoding.GetByteCount("\n");
    byte[] buffer = encoding.GetBytes(lineSeparator);


    using (FileStream fs = new FileStream(path, FileMode.Open)) {
        Int64 lineCount = 0;
        Int64 endPosition = fs.Length / sizeOfChar;

        for (Int64 position = sizeOfChar; position 

The above is the detailed content of How Can I Efficiently Retrieve the Last 10 Lines of a Gigantic Text File?. For more information, please follow other related articles on the PHP Chinese website!