search
HomeBackend DevelopmentPHP TutorialDetailed explanation of how to read large files in PHP_PHP Tutorial
Detailed explanation of how to read large files in PHP_PHP TutorialJul 20, 2016 am 11:11 AM
phpintroduceclassmatebigaccomplishdocumentarticlemethodDetailed explanationread

This article will introduce to you the detailed explanation of the implementation method of reading large files in PHP. Students who need to know more can refer to it for reference.

The requirements are as follows: There is a log file of about 1G with about 5 million lines. Use PHP to return the contents of the last few lines.

Implementation method:

1. Directly use the file function to operate
Note: Since the file function reads all the content into the memory at one time, PHP in order to prevent some poor writing The program takes up too much memory and causes insufficient system memory, causing the server to crash. Therefore, by default, the maximum memory usage is limited to 16M. This is set through memory_limit = 16M in php.ini. If this value is set -1, the memory usage is not limited.

The following is a piece of code that uses file to extract the last line of this file.
The entire code execution takes 116.9613 (s).

The code is as follows Copy code ini_set('memory_limit','-1');
 代码如下 复制代码
ini_set('memory_limit','-1');
$file = 'access.log';
$data = file($file);
$line = $data[count($data)-1];
echo $line;
$file = 'access.log';

$data = file($file);

$line = $data[count( $data)-1];

echo $line;

My machine has 2 G of memory. When I press F5 to run, the system directly It turned gray and recovered after almost 20 minutes. It can be seen that the consequences of reading such a large file directly into the memory are serious, so it is not a last resort. The memory_limit cannot be adjusted too high, otherwise the only choice is to call the computer room. Let the machine reset.

2. Directly call the linux tail command to display the last few lines
 代码如下 复制代码
file = 'access.log';
$file = escapeshellarg($file); // 对命令行参数进行安全转义
$line = `tail -n 1 $file`;
echo $line;

In the linux command line, you can directly use tail -n 10 access.log to easily display For the last few lines of the log file, you can directly use php to call the tail command. The execution of the php code is as follows.

The entire code execution takes 0.0034 (s)

The code is as follows Copy code
file = 'access.log';
$file = escapeshellarg($file); // Safely escape command line parameters
$line = `tail -n 1 $file`;
echo $line;

3. Directly use php’s fseek to perform file operations

 代码如下 复制代码

function tail($fp,$n,$base=5)
{
    assert($n>0);
    $pos = $n+1;
    $lines = array();
    while(count($lines)        try{
            fseek($fp,-$pos,SEEK_END);
        } catch (Exception $e){
            fseek(0);
            break;
        }
        $pos *= $base;
        while(!feof($fp)){
            array_unshift($lines,fgets($fp));
        }
    }
    return array_slice($lines,0,$n);
}
var_dump(tail(fopen("access.log","r+"),10));

This method is the most common method. It does not need to read all the contents of the file into the memory, but It operates directly through pointers, so the efficiency is quite efficient. When using fseek to operate files, there are many different methods, and the efficiency may be slightly different. The following are two commonly used methods. method 1: First find the last EOF of the file through fseek, then find the starting position of the last line, get the data of this line, then find the starting position of the next line, then take the position of this line, and so on. Until the $num row is found. The implementation code is as followsThe entire code execution takes 0.0095 (s)
The code is as follows Copy code
function tail($fp,$n,$base=5){ assert($n>0); $pos = $n+1; $lines = array(); while(count($lines) try{ fseek($fp,-$pos,SEEK_END); } catch (Exception $e){ fseek(0); break; } $pos *= $base; while(!feof($fp)){ array_unshift($lines,fgets($fp)); } } return array_slice($ lines,0,$n);}var_dump(tail(fopen("access.log","r+"),10));

Method 2:

Still use fseek to read from the end of the file, but this time it is not reading bit by bit, but reading piece by piece. Each time a piece of data is read, it will be read The fetched data is placed in a buf, and then the number of newline characters (n) is used to determine whether the last $num rows of data have been read.
The implementation code is as follows
The entire code execution takes 0.0009 (s ).

$t = " ";$data = "";
The code is as follows
 代码如下 复制代码

$fp = fopen($file, "r");
$line = 10;
$pos = -2;
$t = " ";
$data = "";
while ($line > 0) {
    while ($t != "n") {
        fseek($fp, $pos, SEEK_END);
        $t = fgetc($fp);
        $pos --;
    }
    $t = " ";
    $data .= fgets($fp);
    $line --;
}
fclose ($fp);
echo $data

Copy code

$fp = fopen($file, "r");

$line = 10;
 代码如下 复制代码

$fp = fopen($file, "r");
$num = 10;
$chunk = 4096;
$fs = sprintf("%u", filesize($file));
$max = (intval($fs) == PHP_INT_MAX) ? PHP_INT_MAX : filesize($file);
for ($len = 0; $len   $seekSize = ($max - $len > $chunk) ? $chunk : $max - $len;
    fseek($fp, ($len + $seekSize) * -1, SEEK_END);
    $readData = fread($fp, $seekSize) . $readData;
 
    if (substr_count($readData, "n") >= $num + 1) {
        preg_match("!(.*?n){".($num)."}$!", $readData, $match);
        $data = $match[0];
        break;
    }
}
fclose($fp);
echo $data;

$pos = -2;
while ($line > 0) {
while ($t != "n") { fseek($fp, $pos, SEEK_END);

$t = fgetc($fp);

$pos --; $t = " "; $data .= fgets($fp); $line -- ;}fclose ($fp);echo $data Method 3:
The entire code execution takes time to complete 0.0003(s)
The code is as follows
Copy code
$fp = fopen($file, "r");$num = 10;$chunk = 4096;$fs = sprintf("% u", filesize($file));$max = (intval($fs) == PHP_INT_MAX) ? PHP_INT_MAX : filesize($file);for ($len = 0; $len $seekSize = ($max - $len > $chunk) ? $chunk : $max - $len; fseek($fp, ($len + $ seekSize) * -1, SEEK_END); $readData = fread($fp, $seekSize) . $readData; if (substr_count($readData, "n") >= $num + 1) { preg_match("!(.*?n){".($num)."}$!", $readData, $match); $data = $match[0]; break; }}fclose($fp);echo $data; http://www.bkjia.com/PHPjc/444602.htmlwww.bkjia.comtruehttp: //www.bkjia.com/PHPjc/444602.htmlTechArticleThis article will introduce you to the detailed implementation method of reading large files in php. Students who need to know more can Go to Reference. The requirements are as follows: There is a log file of about 1G, about...
Statement
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn
php怎么把负数转为正整数php怎么把负数转为正整数Apr 19, 2022 pm 08:59 PM

php把负数转为正整数的方法:1、使用abs()函数将负数转为正数,使用intval()函数对正数取整,转为正整数,语法“intval(abs($number))”;2、利用“~”位运算符将负数取反加一,语法“~$number + 1”。

php怎么实现几秒后执行一个函数php怎么实现几秒后执行一个函数Apr 24, 2022 pm 01:12 PM

实现方法:1、使用“sleep(延迟秒数)”语句,可延迟执行函数若干秒;2、使用“time_nanosleep(延迟秒数,延迟纳秒数)”语句,可延迟执行函数若干秒和纳秒;3、使用“time_sleep_until(time()+7)”语句。

php怎么除以100保留两位小数php怎么除以100保留两位小数Apr 22, 2022 pm 06:23 PM

php除以100保留两位小数的方法:1、利用“/”运算符进行除法运算,语法“数值 / 100”;2、使用“number_format(除法结果, 2)”或“sprintf("%.2f",除法结果)”语句进行四舍五入的处理值,并保留两位小数。

php怎么根据年月日判断是一年的第几天php怎么根据年月日判断是一年的第几天Apr 22, 2022 pm 05:02 PM

判断方法:1、使用“strtotime("年-月-日")”语句将给定的年月日转换为时间戳格式;2、用“date("z",时间戳)+1”语句计算指定时间戳是一年的第几天。date()返回的天数是从0开始计算的,因此真实天数需要在此基础上加1。

php怎么替换nbsp空格符php怎么替换nbsp空格符Apr 24, 2022 pm 02:55 PM

方法:1、用“str_replace(" ","其他字符",$str)”语句,可将nbsp符替换为其他字符;2、用“preg_replace("/(\s|\&nbsp\;||\xc2\xa0)/","其他字符",$str)”语句。

php怎么判断有没有小数点php怎么判断有没有小数点Apr 20, 2022 pm 08:12 PM

php判断有没有小数点的方法:1、使用“strpos(数字字符串,'.')”语法,如果返回小数点在字符串中第一次出现的位置,则有小数点;2、使用“strrpos(数字字符串,'.')”语句,如果返回小数点在字符串中最后一次出现的位置,则有。

php怎么设置implode没有分隔符php怎么设置implode没有分隔符Apr 18, 2022 pm 05:39 PM

在PHP中,可以利用implode()函数的第一个参数来设置没有分隔符,该函数的第一个参数用于规定数组元素之间放置的内容,默认是空字符串,也可将第一个参数设置为空,语法为“implode(数组)”或者“implode("",数组)”。

php怎么读取字符串后几个字符php怎么读取字符串后几个字符Apr 22, 2022 pm 08:31 PM

在php中,可以使用substr()函数来读取字符串后几个字符,只需要将该函数的第二个参数设置为负值,第三个参数省略即可;语法为“substr(字符串,-n)”,表示读取从字符串结尾处向前数第n个字符开始,直到字符串结尾的全部字符。

See all articles

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

AI Hentai Generator

AI Hentai Generator

Generate AI Hentai for free.

Hot Article

R.E.P.O. Energy Crystals Explained and What They Do (Yellow Crystal)
2 weeks agoBy尊渡假赌尊渡假赌尊渡假赌
Repo: How To Revive Teammates
4 weeks agoBy尊渡假赌尊渡假赌尊渡假赌
Hello Kitty Island Adventure: How To Get Giant Seeds
4 weeks agoBy尊渡假赌尊渡假赌尊渡假赌

Hot Tools

EditPlus Chinese cracked version

EditPlus Chinese cracked version

Small size, syntax highlighting, does not support code prompt function

Dreamweaver Mac version

Dreamweaver Mac version

Visual web development tools

ZendStudio 13.5.1 Mac

ZendStudio 13.5.1 Mac

Powerful PHP integrated development environment

SublimeText3 Mac version

SublimeText3 Mac version

God-level code editing software (SublimeText3)

mPDF

mPDF

mPDF is a PHP library that can generate PDF files from UTF-8 encoded HTML. The original author, Ian Back, wrote mPDF to output PDF files "on the fly" from his website and handle different languages. It is slower than original scripts like HTML2FPDF and produces larger files when using Unicode fonts, but supports CSS styles etc. and has a lot of enhancements. Supports almost all languages, including RTL (Arabic and Hebrew) and CJK (Chinese, Japanese and Korean). Supports nested block-level elements (such as P, DIV),