Reading large files has always been a headache. If we use PHP to develop and read small files, we can directly use various functions to achieve it. However, once we get to a large article, we will find that the commonly used methods cannot be used normally or take too long. It's too stuck. Let's take a look at the solution to the problem of reading large files in PHP. I hope it will be helpful to everyone.
In PHP, when reading files, the fastest way is to use some functions such as file and file_get_contents. It can be completed beautifully with just a few lines of code. The functionality we need. But when the file being operated is a relatively large file, these functions may be insufficient. The following will start with a requirement to explain the commonly used operating methods for reading large files.
Requirements:
There is an 800M log file with approximately more than 5 million lines. Use PHP to return the contents of the last few lines.
Implementation method:
1. Directly use the file function to operate
Since the file function reads all the contents into the memory at one time, PHP in order to prevent some poorly written programs Occupying too much memory will cause insufficient system memory and cause the server to crash. Therefore, by default, the maximum memory usage is limited to 16M. This is set through memory_limit = 16M in php.ini. If this value is set to -1 , the memory usage is unlimited.
The following is a piece of code that uses file to extract the last line of this file:
<?php ini_set('memory_limit', '-1'); $file = 'access.log'; $data = file($file); $line = $data[count($data) - 1]; echo $line; ?>
The entire code execution takes 116.9613 (s).
My machine has 2 G of memory. When I press F5 to run, the system turns gray and recovers after almost 20 minutes. It can be seen that the consequences of reading such a large file directly into the memory are serious, so As a last resort, memory_limit cannot be set too high, otherwise you will have to call the computer room to reset the machine.
2. Directly call the Linux tail command to display the last few lines
Under the Linux command line, you can directly use tail -n 10 access.log to easily display the last few lines of the log file. You can directly Use PHP to call the tail command and execute the PHP code as follows:
<?php $file = 'access.log'; $file = escapeshellarg($file); // 对命令行参数进行安全转义 $line = `tail -n 1 $file`; echo $line; ?>
The entire code execution takes 0.0034 (s)
3. Directly use PHP’s fseek to perform file operations
This This method is the most common method. It does not need to read all the contents of the file, but operates directly through pointers, so the efficiency is quite efficient. When using fseek to operate files, there are many different methods, and the efficiency may be slightly different. The following are two commonly used methods:
Method 1
First find the file through fseek The last EOF, then find the starting position of the last row, get the data of this row, then find the starting position of the next row, then take the position of this row, and so on, until the $num row is found.
The implementation code is as follows
<?php $fp = fopen($file, "r"); $line = 10; $pos = -2; $t = " "; $data = ""; while ($line > 0) { while ($t != "\n") { fseek($fp, $pos, SEEK_END); $t = fgetc($fp); $pos--; } $t = " "; $data .= fgets($fp); $line--; } fclose($fp); echo $data ?>
It takes 0.0095 (s) to complete the execution of the entire code
Method 2
Still use fseek to read from the end of the file, but this time it is not the same Read bit by bit, but read piece by piece. Every time a piece of data is read, the read data is placed in a buf, and then the number of newline characters (\n) is used to determine whether the last piece of data has been read. $num rows of data.
The implementation code is as follows
<?php $fp = fopen($file, "r"); $num = 10; $chunk = 4096; $fs = sprintf("%u", filesize($file)); $max = (intval($fs) == PHP_INT_MAX) ? PHP_INT_MAX : filesize($file); for ($len = 0; $len < $max; $len += $chunk) { $seekSize = ($max - $len > $chunk) ? $chunk : $max - $len; fseek($fp, ($len + $seekSize) * -1, SEEK_END); $readData = fread($fp, $seekSize) . $readData; if (substr_count($readData, "\n") >= $num + 1) { preg_match("!(.*?\n){" . ($num) . "}$!", $readData, $match); $data = $match[0]; break; } } fclose($fp); echo $data; ?>
The entire code execution takes 0.0009(s).
Method 3
<?php function tail($fp, $n, $base = 5) { assert($n > 0); $pos = $n + 1; $lines = array(); while (count($lines) <= $n) { try { fseek($fp, -$pos, SEEK_END); } catch (Exception $e) { fseek(0); break; } $pos *= $base; while (!feof($fp)) { array_unshift($lines, fgets($fp)); } } return array_slice($lines, 0, $n); } var_dump(tail(fopen("access.log", "r+"), 10)); ?>
The entire code execution takes 0.0003(s)
Method 4, PHP’s stream_get_line function, reads quickly, and reads 500,000 pieces of data file, it will take about 20 seconds! The example code is as follows
$fp = fopen('./iis.log', 'r'); //文件 while (!feof($fp)) { //for($j=1;$j<=1000;$j++) { //读取下面的1000行并存储到数组中 $logarray[] = stream_get_line($fp, 65535, "\n"); // break; // } }
Related recommendations:
php Detailed explanation of file reading series methods
PHP file reading fread, fgets, fgetc, file_get_contents and file function usage example code
##A brief introduction to PHP file lock and process lock
The above is the detailed content of Example comparison of how PHP reads large files efficiently. For more information, please follow other related articles on the PHP Chinese website!

php把负数转为正整数的方法:1、使用abs()函数将负数转为正数,使用intval()函数对正数取整,转为正整数,语法“intval(abs($number))”;2、利用“~”位运算符将负数取反加一,语法“~$number + 1”。

实现方法:1、使用“sleep(延迟秒数)”语句,可延迟执行函数若干秒;2、使用“time_nanosleep(延迟秒数,延迟纳秒数)”语句,可延迟执行函数若干秒和纳秒;3、使用“time_sleep_until(time()+7)”语句。

php除以100保留两位小数的方法:1、利用“/”运算符进行除法运算,语法“数值 / 100”;2、使用“number_format(除法结果, 2)”或“sprintf("%.2f",除法结果)”语句进行四舍五入的处理值,并保留两位小数。

判断方法:1、使用“strtotime("年-月-日")”语句将给定的年月日转换为时间戳格式;2、用“date("z",时间戳)+1”语句计算指定时间戳是一年的第几天。date()返回的天数是从0开始计算的,因此真实天数需要在此基础上加1。

方法:1、用“str_replace(" ","其他字符",$str)”语句,可将nbsp符替换为其他字符;2、用“preg_replace("/(\s|\ \;||\xc2\xa0)/","其他字符",$str)”语句。

php判断有没有小数点的方法:1、使用“strpos(数字字符串,'.')”语法,如果返回小数点在字符串中第一次出现的位置,则有小数点;2、使用“strrpos(数字字符串,'.')”语句,如果返回小数点在字符串中最后一次出现的位置,则有。

在PHP中,可以利用implode()函数的第一个参数来设置没有分隔符,该函数的第一个参数用于规定数组元素之间放置的内容,默认是空字符串,也可将第一个参数设置为空,语法为“implode(数组)”或者“implode("",数组)”。

php字符串有下标。在PHP中,下标不仅可以应用于数组和对象,还可应用于字符串,利用字符串的下标和中括号“[]”可以访问指定索引位置的字符,并对该字符进行读写,语法“字符串名[下标值]”;字符串的下标值(索引值)只能是整数类型,起始值为0。


Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

AI Hentai Generator
Generate AI Hentai for free.

Hot Article

Hot Tools

SublimeText3 Linux new version
SublimeText3 Linux latest version

Notepad++7.3.1
Easy-to-use and free code editor

Atom editor mac version download
The most popular open source editor

WebStorm Mac version
Useful JavaScript development tools

ZendStudio 13.5.1 Mac
Powerful PHP integrated development environment
