search
HomeBackend DevelopmentPHP TutorialExample comparison of how PHP reads large files efficiently

Example comparison of how PHP reads large files efficiently

Dec 29, 2017 pm 06:59 PM
phpExampledocument

Reading large files has always been a headache. If we use PHP to develop and read small files, we can directly use various functions to achieve it. However, once we get to a large article, we will find that the commonly used methods cannot be used normally or take too long. It's too stuck. Let's take a look at the solution to the problem of reading large files in PHP. I hope it will be helpful to everyone.

In PHP, when reading files, the fastest way is to use some functions such as file and file_get_contents. It can be completed beautifully with just a few lines of code. The functionality we need. But when the file being operated is a relatively large file, these functions may be insufficient. The following will start with a requirement to explain the commonly used operating methods for reading large files.

Requirements:
There is an 800M log file with approximately more than 5 million lines. Use PHP to return the contents of the last few lines.

Implementation method:

1. Directly use the file function to operate
Since the file function reads all the contents into the memory at one time, PHP in order to prevent some poorly written programs Occupying too much memory will cause insufficient system memory and cause the server to crash. Therefore, by default, the maximum memory usage is limited to 16M. This is set through memory_limit = 16M in php.ini. If this value is set to -1 , the memory usage is unlimited.

The following is a piece of code that uses file to extract the last line of this file:

<?php
  ini_set(&#39;memory_limit&#39;, &#39;-1&#39;);
  $file = &#39;access.log&#39;;
  $data = file($file);
  $line = $data[count($data) - 1];
  echo $line;
?>

The entire code execution takes 116.9613 (s).
My machine has 2 G of memory. When I press F5 to run, the system turns gray and recovers after almost 20 minutes. It can be seen that the consequences of reading such a large file directly into the memory are serious, so As a last resort, memory_limit cannot be set too high, otherwise you will have to call the computer room to reset the machine.

2. Directly call the Linux tail command to display the last few lines
Under the Linux command line, you can directly use tail -n 10 access.log to easily display the last few lines of the log file. You can directly Use PHP to call the tail command and execute the PHP code as follows:

<?php
  $file = &#39;access.log&#39;;
  $file = escapeshellarg($file); // 对命令行参数进行安全转义
  $line = `tail -n 1 $file`;
  echo $line;
?>

The entire code execution takes 0.0034 (s)

3. Directly use PHP’s fseek to perform file operations
This This method is the most common method. It does not need to read all the contents of the file, but operates directly through pointers, so the efficiency is quite efficient. When using fseek to operate files, there are many different methods, and the efficiency may be slightly different. The following are two commonly used methods:

Method 1
First find the file through fseek The last EOF, then find the starting position of the last row, get the data of this row, then find the starting position of the next row, then take the position of this row, and so on, until the $num row is found.
The implementation code is as follows

<?php
$fp = fopen($file, "r");
$line = 10;
$pos = -2;
$t = " ";
$data = "";
while ($line > 0)
{
 while ($t != "\n")
 {
 fseek($fp, $pos, SEEK_END);
 $t = fgetc($fp);
 $pos--;
 }
 $t = " ";
 $data .= fgets($fp);
 $line--;
}
fclose($fp);
echo $data
?>

It takes 0.0095 (s) to complete the execution of the entire code

Method 2
Still use fseek to read from the end of the file, but this time it is not the same Read bit by bit, but read piece by piece. Every time a piece of data is read, the read data is placed in a buf, and then the number of newline characters (\n) is used to determine whether the last piece of data has been read. $num rows of data.
The implementation code is as follows

<?php
$fp = fopen($file, "r");
$num = 10;
$chunk = 4096;
$fs = sprintf("%u", filesize($file));
$max = (intval($fs) == PHP_INT_MAX) ? PHP_INT_MAX : filesize($file);
for ($len = 0; $len < $max; $len += $chunk)
{
 $seekSize = ($max - $len > $chunk) ? $chunk : $max - $len;
 fseek($fp, ($len + $seekSize) * -1, SEEK_END);
 $readData = fread($fp, $seekSize) . $readData;
 if (substr_count($readData, "\n") >= $num + 1)
 {
 preg_match("!(.*?\n){" . ($num) . "}$!", $readData, $match);
 $data = $match[0];
 break;
 }
}
fclose($fp);
echo $data;
?>

The entire code execution takes 0.0009(s).

Method 3

<?php
function tail($fp, $n, $base = 5)
{
 assert($n > 0);
 $pos = $n + 1;
 $lines = array();
 while (count($lines) <= $n)
 {
 try
 {
  fseek($fp, -$pos, SEEK_END);
 }
 catch (Exception $e)
 {
  fseek(0);
  break;
 }
 $pos *= $base;
 while (!feof($fp))
 {
  array_unshift($lines, fgets($fp));
 }
 }
 return array_slice($lines, 0, $n);
}
var_dump(tail(fopen("access.log", "r+"), 10));
?>

The entire code execution takes 0.0003(s)

Method 4, PHP’s stream_get_line function, reads quickly, and reads 500,000 pieces of data file, it will take about 20 seconds! The example code is as follows

$fp = fopen(&#39;./iis.log&#39;, &#39;r&#39;); //文件 
while (!feof($fp)) { 
 //for($j=1;$j<=1000;$j++) {     //读取下面的1000行并存储到数组中 
 $logarray[] = stream_get_line($fp, 65535, "\n"); 
    // break;
 // } 
 
 }

Related recommendations:

php Detailed explanation of file reading series methods

PHP file reading fread, fgets, fgetc, file_get_contents and file function usage example code

##A brief introduction to PHP file lock and process lock

The above is the detailed content of Example comparison of how PHP reads large files efficiently. For more information, please follow other related articles on the PHP Chinese website!

Statement
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn
Explain how load balancing affects session management and how to address it.Explain how load balancing affects session management and how to address it.Apr 29, 2025 am 12:42 AM

Load balancing affects session management, but can be resolved with session replication, session stickiness, and centralized session storage. 1. Session Replication Copy session data between servers. 2. Session stickiness directs user requests to the same server. 3. Centralized session storage uses independent servers such as Redis to store session data to ensure data sharing.

Explain the concept of session locking.Explain the concept of session locking.Apr 29, 2025 am 12:39 AM

Sessionlockingisatechniqueusedtoensureauser'ssessionremainsexclusivetooneuseratatime.Itiscrucialforpreventingdatacorruptionandsecuritybreachesinmulti-userapplications.Sessionlockingisimplementedusingserver-sidelockingmechanisms,suchasReentrantLockinJ

Are there any alternatives to PHP sessions?Are there any alternatives to PHP sessions?Apr 29, 2025 am 12:36 AM

Alternatives to PHP sessions include Cookies, Token-based Authentication, Database-based Sessions, and Redis/Memcached. 1.Cookies manage sessions by storing data on the client, which is simple but low in security. 2.Token-based Authentication uses tokens to verify users, which is highly secure but requires additional logic. 3.Database-basedSessions stores data in the database, which has good scalability but may affect performance. 4. Redis/Memcached uses distributed cache to improve performance and scalability, but requires additional matching

Define the term 'session hijacking' in the context of PHP.Define the term 'session hijacking' in the context of PHP.Apr 29, 2025 am 12:33 AM

Sessionhijacking refers to an attacker impersonating a user by obtaining the user's sessionID. Prevention methods include: 1) encrypting communication using HTTPS; 2) verifying the source of the sessionID; 3) using a secure sessionID generation algorithm; 4) regularly updating the sessionID.

What is the full form of PHP?What is the full form of PHP?Apr 28, 2025 pm 04:58 PM

The article discusses PHP, detailing its full form, main uses in web development, comparison with Python and Java, and its ease of learning for beginners.

How does PHP handle form data?How does PHP handle form data?Apr 28, 2025 pm 04:57 PM

PHP handles form data using $\_POST and $\_GET superglobals, with security ensured through validation, sanitization, and secure database interactions.

What is the difference between PHP and ASP.NET?What is the difference between PHP and ASP.NET?Apr 28, 2025 pm 04:56 PM

The article compares PHP and ASP.NET, focusing on their suitability for large-scale web applications, performance differences, and security features. Both are viable for large projects, but PHP is open-source and platform-independent, while ASP.NET,

Is PHP a case-sensitive language?Is PHP a case-sensitive language?Apr 28, 2025 pm 04:55 PM

PHP's case sensitivity varies: functions are insensitive, while variables and classes are sensitive. Best practices include consistent naming and using case-insensitive functions for comparisons.

See all articles

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

Video Face Swap

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Tools

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment

WebStorm Mac version

WebStorm Mac version

Useful JavaScript development tools

SAP NetWeaver Server Adapter for Eclipse

SAP NetWeaver Server Adapter for Eclipse

Integrate Eclipse with SAP NetWeaver application server.

Safe Exam Browser

Safe Exam Browser

Safe Exam Browser is a secure browser environment for taking online exams securely. This software turns any computer into a secure workstation. It controls access to any utility and prevents students from using unauthorized resources.

mPDF

mPDF

mPDF is a PHP library that can generate PDF files from UTF-8 encoded HTML. The original author, Ian Back, wrote mPDF to output PDF files "on the fly" from his website and handle different languages. It is slower than original scripts like HTML2FPDF and produces larger files when using Unicode fonts, but supports CSS styles etc. and has a lot of enhancements. Supports almost all languages, including RTL (Arabic and Hebrew) and CJK (Chinese, Japanese and Korean). Supports nested block-level elements (such as P, DIV),