Let’s talk about thoughts about PHP memory overflow

When doing large-scale data export and data import recently, I often encounter the problem of PHP memory overflow. After solving the problem, I summarized some experiences and compiled them into articles to record. .

Optimization points

1. Optimize SQL statements, avoid slow queries, build indexes reasonably, and query specified fields. SQL optimization will not be discussed here.

2. When the query result set is a large object, it is converted to an array. There are generally methods in the framework to convert, such as toArray() in Laravel and asArray() in Yii2.

3. For data cutting of large arrays, PHP functions include array_chunk() and array_slice().

4. For large strings and objects, use &.

5. Used variables are unset in time.

6. The exported file format is changed from excel to csv

7. ini_set('memory_limit',''), set the memory that the program can use ( Not recommended).

Thinking

Memory Management

How to manage the memory of PHP? When learning C language, developers need to manually manage memory. In PHP, the Zend engine provides a special memory manager for processing request-related data. Requesting related data only requires serving a single request, and the data will be released at the end of the request at the latest.

The above picture is a description screenshot from the official website

Preventing memory leaks and releasing all memory as quickly as possible is an important part of memory management. For security reasons, Zend Engine will release all memory allocated by the API lock mentioned above.

Garbage collection mechanism

A brief explanation:

Before PHP5.3, reference counting was used for management. Variables in PHP are stored in the variable container of zval. When a variable is referenced, the reference count is 1. When the variable reference count is 0, PHP will destroy the variable in memory. However, when the reference count is cyclically referenced, the reference count will not be reduced to 0, causing a memory leak.

After PHP5.3, optimization has been done. It does not enter the recycling cycle every time the reference count decreases. Garbage collection starts only after the root buffer is full. This can solve the problem of circular references and can also restore the total Memory leaks remain below a threshold.

Code

Since memory overflow is often encountered when using phpexcel, here is a code to generate a csv file:

<?php

namespace api\service;

class ExportService
{

    public static $outPutFile = &#39;&#39;;

    /**
     * 导出文件
     * @param string $fileName
     * @param $data
     * @param array $formFields
     * @return mixed
     */
    public static function exportData($fileName = &#39;&#39;, $data, $formFields = [])
    {
        $fileArr = [];
        $tmpPath = \Yii::$app->params[&#39;excelSavePath&#39;];

        foreach (array_chunk($data, 10000) as $key => $value) {
            self::$outPutFile = &#39;&#39;;
            $subject          = !empty($fileName) ? $fileName : &#39;data_&#39;;
            $subject          .= date(&#39;YmdHis&#39;);
            if (empty($value) || empty($formFields)) {
                continue;
            }

            self::$outPutFile = $tmpPath . $subject . $key . &#39;.csv&#39;;
            if (!file_exists(self::$outPutFile)) {
                touch(self::$outPutFile);
            }
            $index  = array_keys($formFields);
            $header = array_values($formFields);
            self::outPut($header);

            foreach ($value as $k => $v) {
                $tmpData = [];
                foreach ($index as $item) {
                    $tmpData[] = isset($v[$item]) ? $v[$item] : &#39;&#39;;
                }
                self::outPut($tmpData);
            }
            $fileArr[] = self::$outPutFile;
        }
        
        $zipFile = $tmpPath . $fileName . date(&#39;YmdHi&#39;) . &#39;.zip&#39;;
        $zipRes = self::zipFile($fileArr, $zipFile);
        return $zipRes;
    }

    /**
     * 向文件写入数据
     * @param array $data
     */
    public static function outPut($data = [])
    {
        if (is_array($data) && !empty($data)) {
            $data = implode(&#39;,&#39;, $data);
            file_put_contents(self::$outPutFile, iconv("UTF-8", "GB2312//IGNORE", $data) . PHP_EOL, FILE_APPEND);
        }
    }

    /**
     * 压缩文件
     * @param $sourceFile
     * @param $distFile
     * @return mixed
     */
    public static function zipFile($sourceFile, $distFile)
    {
        $zip = new \ZipArchive();
        if ($zip->open($distFile, \ZipArchive::CREATE) !== true) {
            return $sourceFile;
        }

        $zip->open($distFile, \ZipArchive::CREATE);
        foreach ($sourceFile as $file) {
            $fileContent = file_get_contents($file);
            $file        = iconv(&#39;utf-8&#39;, &#39;GBK&#39;, basename($file));
            $zip->addFromString($file, $fileContent);
        }
        $zip->close();
        return $distFile;
    }
    
        /**
     * 下载文件
     * @param $filePath
     * @param $fileName
     */
    public static function download($filePath, $fileName)
    {
        if (!file_exists($filePath . $fileName)) {
            header(&#39;HTTP/1.1 404 NOT FOUND&#39;);
        } else {
            //以只读和二进制模式打开文件
            $file = fopen($filePath . $fileName, "rb");

            //告诉浏览器这是一个文件流格式的文件
            Header("Content-type: application/octet-stream");
            //请求范围的度量单位
            Header("Accept-Ranges: bytes");
            //Content-Length是指定包含于请求或响应中数据的字节长度
            Header("Accept-Length: " . filesize($filePath . $fileName));
            //用来告诉浏览器，文件是可以当做附件被下载，下载后的文件名称为$file_name该变量的值
            Header("Content-Disposition: attachment; filename=" . $fileName);

            //读取文件内容并直接输出到浏览器
            echo fread($file, filesize($filePath . $fileName));
            fclose($file);
            exit();
        }
    }
}

Calling out the code

$fileName = "库存导入模板";
$stockRes = []; // 导出的数据
$formFields = [
    'store_id'  => '门店ID',
    'storeName' => '门店名称',
    'sku'       => 'SKU编码',
    'name'      => 'SKU名称',
    'stock'     => '库存',
    'reason'    => '原因'
];
$fileRes    = ExportService::exportData($fileName, $stockRes, $formFields);
$tmpPath    = \Yii::$app->params['excelSavePath']; // 文件路径
$fileName   = str_replace($tmpPath, '', $fileRes);

// 下载文件
ExportService::download($tmpPath, $fileName);

Recommended learning: PHP video tutorial

The above is the detailed content of Let’s talk about thoughts about PHP memory overflow. For more information, please follow other related articles on the PHP Chinese website!