PHP handles memory limit problem of extremely long data

Question

Currently, there are multiple radar devices that scan a certain area. After scanning, data of approximately 1.4 million points (each point contains 4 attributes) will be returned.
Then the radar device splices the data into a string separated by ,: "x1,y1,a1,p1,x2,y2,a2,p2......x1400000, y1400000,a1400000,p1400000"
The size of this string should be tens of megabytes, less than 100 megabytes.
PHP code is processed like this:

$num=$_POST['num'];//点的总数量
$str=$_POST['str'];//点的4个属性组成的字符串
$str_arr=expload(',',$str);//将字符串转换为数组

//将数组转换成json字符串
$point=array();
for($n=1;$n<=$num;$n++){
    $x=$str_arr[$n*4-4];
    $y=$str_arr[$n*4-3];
    $a=$str_arr[$n*4-2];
    $p=$str_arr[$n*4-1];
    $point[]=array("x"=>$x,"y"=>$y,"a"=>$a,"$p"=>$p);
}
$json_str=json_encode($point);

//将json字符串保存在".txt"文件中
$handle=fopen("./1.txt","w");
fwrite($handle, $json_str);
fclose($handle);

//返回请求成功信息
echo "ask=1&cmd=3";
exit;

I don’t know much about memory consumption, and this is the first time to process such a large amount of data.
When the data transmitted by the radar is only 11W groups of points (about 3MB data) , I can successfully receive and process it here
When the data is a little larger, it will promptThe memory limit is 128M
I checked some methods on the Internet, most of them are called changesphp.iniIn the filememory_limit=-1
But after changing it, it still cannot be transmitted
Currently, the discussion with Radar is that he can transmit it to me at most every time 10WGroup points, and then add one more parameterHow many points are left in this transmission. After PHP receives it, it will be given to him as a return value. He will call back and continue to send it to me, but after calculating it like this, It takes a total of 14 to 15 transmissions to send a set of data...and the feasibility is not yet certain
Could you please tell me how you dealt with this problem when you encountered it during development?
Waiting online. . .

Answer 1

ini_set('memory_limit', '500M'); //内存限制
set_time_limit(0); //执行超时，0代表无限等待

Answer 2

Question: Is gzip enabled? This kind of pure JSON data has a high compression rate after it is turned on. After changing the memory limit, the network connection may also time out.

Answer 3

1. There is a length limit for the body received by the web server. If it is too long, it cannot be accepted, such as nginx-client_max_body_size

2. Batch transmission is a better solution in web mode

3. A better way is to use socket binary transmission, which does not have various restrictions in web mode and is more efficient

Answer 4

First of all, your string takes up a lot of memory. If you convert it to an array, it will take up a lot of memory.

Depending on your needs, the final step is to convert it to json and save it to a file. Json is also a string, so why not process it directly as a string.

If it is php7, you cannot use preg_replace and need to switch to preg_replace_callback to handle it

$str=$_POST['str'];//点的4个属性组成的字符串
$json_str = '['.substr(preg_replace('/([^,]+),([^,]+),([^,]+),([^,]+),/iU','{"x":"","y":"","a":"","p":""},',$str.','),0,-1).']';
//将json字符串保存在```.txt```文件中
$handle=fopen("./1.txt","w");
fwrite($handle, $json_str);
fclose($handle);

Answer 5

Enlarge the memory limit and increase the data volume of the post. . Basically, it’s no problem to handle a few hundred megabytes