How Can I Efficiently Retrieve the Last 10 Lines of a Gigantic Text File?

Efficiently Retrieving the Last 10 Lines of a Massive Text File

In the realm of large-scale text processing, retrieving the final lines of an exceptionally large file poses a unique challenge. One such file, exceeding 10GB in size, presents a significant hurdle in efficiently obtaining this data.

To tackle this issue, an effective approach involves traversing the file backward, beginning from the end. We aim to locate ten consecutive newlines, indicating the presence of the desired lines. Subsequently, we read forward to capture these lines, considering potential encoding variations.

For instance, in C#, a comprehensive implementation handles cases where the file contains fewer than ten lines. The following code snippet exemplifies this approach:

public static string ReadEndLines(string path, Int64 numberOfLines, Encoding encoding, string lineSeparator) {

    int sizeOfChar = encoding.GetByteCount("\n");
    byte[] buffer = encoding.GetBytes(lineSeparator);


    using (FileStream fs = new FileStream(path, FileMode.Open)) {
        Int64 lineCount = 0;
        Int64 endPosition = fs.Length / sizeOfChar;

        for (Int64 position = sizeOfChar; position < endPosition; position += sizeOfChar) {
            fs.Seek(-position, SeekOrigin.End);
            fs.Read(buffer, 0, buffer.Length);

            if (encoding.GetString(buffer) == lineSeparator) {
                lineCount++;
                if (lineCount == numberOfLines) {
                    byte[] returnBuffer = new byte[fs.Length - fs.Position];
                    fs.Read(returnBuffer, 0, returnBuffer.Length);
                    return encoding.GetString(returnBuffer);
                }
            }
        }

        // handle case where number of lines in file is less than numberOfLines
        fs.Seek(0, SeekOrigin.Begin);
        buffer = new byte[fs.Length];
        fs.Read(buffer, 0, buffer.Length);
        return encoding.GetString(buffer);
    }
}

The above is the detailed content of How Can I Efficiently Retrieve the Last 10 Lines of a Gigantic Text File?. For more information, please follow other related articles on the PHP Chinese website!