Home > Article > Backend Development > PHP batch processing, processing big data, long-term processing_PHP tutorial
PHP batch processing, processing big data, long-term processing_PHP tutorial
- WBOYOriginal
- 2016-07-13 17:52:271056browse
I am currently working on something, written in PHP, because I need to read the name of the image in the database, then access the corresponding image, read the size and resolution of the image, and re-store it into the database. The data volume is now only 5,000. If no settings are added, the default maximum execution time of PHP is 30 seconds.
So within 30 seconds, I can only process 700 items. At the beginning, I can only process 700 or 700 items manually. Later, I felt that this would not work. In the future, the amount of data would exceed 10W, which would exhaust me to death.
After searching online, there are three ways to deal with it.
The first type
It’s very straightforward. Since you default to 30 seconds, I will set your default time to be longer, or there will be no limit.
Add the php ini_set('max_execution_time', '100'); function in php. 100 represents 100 seconds. If you set 0, it means there is no limit. (ps: I use a combination of zend. When it is set to 0, it is not unlimited, but about 120. My solution is to set this value so big)
The second type
I like it very much, it uses a combination of php and batch processing.
[php]
@REM
@REM =='
@SET PHPCLI="D:Program FilesZendZend Studio - 8.0.1pluginsorg.zend.php.debug.debugger.win32.x86_5.3.18.v20110322resourcesphp53php.exe"
@REM PHPCLI Set the path of your php.exe
@%PHPCLI% %0
@goto :EOF
@REM';?>
include'C:\Program Files\Zend\Apache2\htdocs\pic_gather\pic_operate.php';
echo "rfinish";
ob_flush();
exec("pause");
?>
It is a batch processing command, so I spent an hour looking at the batch processing command. Roughly translate the above program.
@: Do not display the following instructions. If removed, the instructions will be printed in the command window
REM is a comment for batch processing, and what follows is a comment and will not be executed. Different from @, @ only means that the command is not printed out and does not represent a comment.
The program starts executing, first bat is run
[php]
@SET PHPCLI="D:Program FilesZendZend Studio - 8.0.1pluginsorg.zend.php.debug.debugger.win32.x86_5.3.18.v20110322resourcesphp53php.exe"
Set the value of PHPCLI
[php] view plaincopy
@%PHPCLI% %0
%PHPCLI% represents the path you set above. Here it means executing php.exe, and then %0 represents the path where the .bat file is located. The combination means that php.exe is used to run the .bat file.
The above phenomenon will occur. PHP does not think REM is a comment, it only thinks it is an ordinary character.
will print @REM first, then
[php]
@REM =='
@SET PHPCLI="D:Program FilesZendZend Studio - 8.0.1pluginsorg.zend.php.debug.debugger.win32.x86_5.3.18.v20110322resourcesphp53php.exe"
@REM PHPCLI Set the path of your php.exe
@%PHPCLI% %0
@goto :EOF
@REM';?>
Here, is php.exe recognized? Does it mean the above? Can @ be recognized in php? This means suppressing errors, then the content in the string '' is == with REM, then there is an error suppressed by @. Get through it in peace.
Go to the next paragraph
[php]
include'C:\Program Files\Zend\Apache2\htdocs\pic_gather\pic_operate.php';
echo "rfinish";
ob_flush();
exec("pause");
?>
Here is the program you want to execute. I don’t want to write it here, so I’ll include it so that .bat can easily become a template.
Your php.exe is now finished, but it’s not finished yet. The batch command just arrived
[php]
@%PHPCLI% %0
Later, in order to prevent the batch command from not recognizing the PHP language, I used
[php]
@goto :EOF
to end the program.
Someone asked why not just do this:
[php]
@SET PHPCLI="D:Program FilesZendZend Studio - 8.0.1pluginsorg.zend.php.debug.debugger.win32.x86_5.3.18.v20110322resourcesphp53php.exe"
@REM PHPCLI
@%PHPCLI% %0
@goto :EOF
Include'C:\Program Files\Zend\Apache2\htdocs\pic_gather\pic_operate.php';
echo "rfinish";
ob_flush();
exec("pause");
?>
It doesn’t need to be so complicated, I tried it too,
It is found that the cmd command line will appear:
@SET PHPCLI="D:Program FilesZendZend Studio - 8.0.1pluginsorg.zend.php.debu
g.debugger.win32.x86_5.3.18.v20110322resourcesphp53php.exe"
@REM PHPCLI
@%PHPCLI% %0
@goto :EOF
The original author of this code just wanted to avoid this situation and print out too much junk information.
So this is the insight of the original author.
The third type
If you do this, you only need to update the string information inside. For example, if you set the class class, if class='natural scenery', then add pic_gather/nature/image in front of the image. Then there is no need to use such complicated things as above, just use sql language directly. The concat() function has such a function.
Author: wolinxuebin