How to solve the problem of memory exhaustion when looping large amounts of data in PHP

Related learning recommendations: php programming(Video)

I recently encountered the following error when developing a PHP program:

PHP Fatal error: Allowed memory size of 268 435 456 bytes exhausted

The error message shows that the maximum allowed memory has been exhausted. I was surprised to encounter such an error at first, but after thinking about it, it is not surprising, because the program I am developing uses a foreach loop statement in a table with 40,000 records. Search the entire table for data with specific characteristics, that is to say, 40,000 pieces of data must be taken out at a time, and then the daily data must be checked one by one. It is conceivable that if all 40,000 pieces of data are loaded into the memory, it would be strange if the memory does not burst.

After all these years of programming, I vaguely remember that PHP provides an API that does not load data all at once. It is a query method that can be used and lost like streaming media, and the data does not accumulate in the memory. . After a simple search, I found the correct usage on the official website.

This problem is called Buffered and Unbuffered queries on PHP's official website. PHP's default query mode is buffered mode. In other words, the query data results will be extracted into memory all at once for processing by the PHP program. This gives the PHP program additional functions, such as counting the number of rows, pointing the pointer to a certain row, etc. What's more important is that the program can repeatedly perform secondary queries and filtering operations on the data set. However, the disadvantage of this buffered query mode is that it consumes memory, that is, it trades space for speed.

In contrast, another PHP query mode is a non-buffered query. The database server will return data one by one instead of all at once. The result is that the PHP program consumes less memory, but it increases The pressure on the database server, because the database will wait for PHP to fetch the data until all the data is fetched.

Obviously, the buffered query mode is suitable for small data volume queries, while non-buffered query is suitable for large data volume queries.

Everyone knows about PHP’s buffered mode query. The following example is how to execute the non-buffered query API.

Non-buffered query method one: mysqli

<?php
$mysqli  = new mysqli("localhost", "my_user", "my_password", "world");
$uresult = $mysqli->query("SELECT Name FROM City", MYSQLI_USE_RESULT);

if ($uresult) {
   while ($row = $uresult->fetch_assoc()) {
       echo $row[&#39;Name&#39;] . PHP_EOL;
   }
}
$uresult->close();
?>

Non-buffered query method two: pdo_mysql

<?php
$pdo = new PDO("mysql:host=localhost;dbname=world", &#39;my_user&#39;, &#39;my_pass&#39;);
$pdo->setAttribute(PDO::MYSQL_ATTR_USE_BUFFERED_QUERY, false);

$uresult = $pdo->query("SELECT Name FROM City");
if ($uresult) {
   while ($row = $uresult->fetch(PDO::FETCH_ASSOC)) {
       echo $row[&#39;Name&#39;] . PHP_EOL;
   }
}
?>

Non-buffered query method three: mysql

<?php
$conn = mysql_connect("localhost", "my_user", "my_pass");
$db   = mysql_select_db("world");

$uresult = mysql_unbuffered_query("SELECT Name FROM City");
if ($uresult) {
   while ($row = mysql_fetch_assoc($uresult)) {
       echo $row[&#39;Name&#39;] . PHP_EOL;
   }
}
?>

php中文网Netizens complained:

This thing should use a generator

This way of writing can be opened directly

It is impossible to process big data According to that writing method

PHP Chinese netizen Xiao Chen gave a correct way

<?php

if (!function_exists(&#39;getYieldBigData&#39;)) {
    /**
     * 使用生成器返回生成器对象
     * @param array $data
     * @return Generator
     */
    function getYieldBigData($data = [])
    {
        foreach ($data as $tmp_data) {
            yield $tmp_data;
        }
        unset($tmp_data);
    }
}


if (!function_exists(&#39;foreachBigData&#39;)) {
    /**
     * 循环大量数据使用生成器来制造值
     * @param array $data
     * @return array|false
     */
    function foreachBigData($data = [])
    {
        if (0 == count($data)) {
            return false;
        }
        $tmp = [];
        foreach (getYieldBigData($data) as $v) {
            $tmp[] = $v;
        }
        unset($v);
        return $tmp;
    }
}

//调用方法
/**
 * @var $data array
 */
$data = [];
/**
 * @var $ret array
 */
$ret = foreachBigData($data);

return $ret;

Novices can also read it Got it!

#As above, this method can reach 1 million/time!

This article comes from the php Chinese website php graphic tutorial channel, the original author: God of War Wukong, the second editor of the php Chinese website.

The above is the detailed content of How to solve the problem of memory exhaustion when looping large amounts of data in PHP. For more information, please follow other related articles on the PHP Chinese website!