PHP implementation code for batch collection and downloading of pictures of beautiful women_PHP tutorial

Design Idea

Considering that it is too troublesome to simply collect pictures from a web page, we directly collect its list page, get the URL of the list and then collect them one by one, but use PHP to match the list The URL of the page is too troublesome. There are many invalid URLs on the first list page, which is really a problem for me, a newbie with regular expressions. After looking at the structure of the list page, I decided to use jquery to obtain the URL. The universal selector of jquery has become powerful again.

jquery gets the url, and then ajax passes the url—> Corresponding PHP file, traverses url parameters—-> Single page collection and saving images

jquery program

Copy code The code is as follows:

<script> <BR>$(document).ready(function(){ <BR>var hrefs =''; <BR>$('.f_folder>a' ).each(function(i){ <BR>var href = $('.f_folder:eq('+i+')>a:eq(0)').attr('href'); <BR>if (href!='undefined'){ <BR>hrefs +=href+','; <BR>} <BR>}) <BR>$.getJSON("http://www.****.com/ 365/getimg.php?hrefs="+hrefs+"&callback=?", function(data){ <BR>//alert(data.info); <BR>}); <BR>}); <BR>&lt ;/script> <BR></script>

Here the url is spliced ​​into ',' separated strings to pass the url. Getjson is used for cross-domain needs. For several common issues with getjson, please refer to

PHP collection program

Copy code The code is as follows:

// Grab 365 pictures
error_reporting(E_ALL ^ ​​E_NOTICE);
set_time_limit(0);//Set PHP timeout
/* *
* Get the current time
*/
function getMicrotime() {

list ($usec, $sec) = explode(" ", microtime());
return ((float) $usec + (float) $sec);
}
$stime = getMicrotime();

$callback = $_GET['callback'];
$hrefs = $_GET['hrefs' ];
$urlarray = explode(',',$hrefs);

//Get all images of the specified url
function getimgs($url){
$dirname = basename( $url,".php");
if(!file_exists($dirname)){
mkdir('365/'.$dirname.'');
}
clearstatcache();
$data = file_get_contents($url);
preg_match_all("/(href|src)=(["|']?)([^ "'>]+.(jpg|png|PNG| JPG|gif))2/i", $data, $matches);
//$matches[3] = array_unique($matches[3]);
unset($data);
$ i=0;

if(count($matches[3])>0){
foreach($matches[3] as $k=>$v){
// Simply determine whether it is a standard url, not a relative path
if(substr($v,0,4)=='http'){

$ext = pathinfo($v,PATHINFO_EXTENSION);/ /Image extensions

if(!file_exists('365/'.$dirname.'/'.$k.'.'.$ext)){
file_put_contents('365/'.$dirname .'/'.$k.'.'.$ext,file_get_contents($v));
$i++;
}else{
unset($v);
}
clearstatcache();
}else{
unset($v);
}
}
unset($matches);
return $i;
}
}

foreach($urlarray as $k=>$v){
if($v!=''){
$j +=getimgs($v);
}
}
$etime = getMicrotime();
echo "A total of ".$j." pictures were collected";
echo "Time spent".($etime-$stime)." seconds";

Considering performance issues: the variables used in the getimgs method are unset after use in order to release memory.

Several knowledge points designed

Judge whether it is a standard valid image url
if(substr($v,0,4)=='http') This is just simple Determine whether the matched image URL is a standard URL, because the collected images may have a relative path. Here I will directly give up the collection of this kind of image. Of course, you can also restore this kind of image to a standard image path. There is another The problem is that even in the standard URL format, such a picture may not be able to be collected, because you don’t know if the picture still exists. Maybe the picture URL is invalid. If you want to more strictly judge whether the picture URL is real and valid, you can recommend it. My previous "PHP Methods to Determine whether a Remote URL is Valid" has three methods to verify whether it is a valid URL.

Get the image format

$ext = pathinfo($v,PATHINFO_EXTENSION);//Image extension

The pathinfo method is used here. In summary, there are 7 methods to obtain it. To find the format of the file, recommended article: "Seven Methods for PHP to Determine Image Format"

Download and save locally

file_put_contents('365/'.$dirname.'/'.$k .'.'.$ext,file_get_contents($v));
file_put_contents() function writes a string to a file.
The same function as calling fopen(), fwrite() and fclose() in sequence.
The file_get_contents() function reads the entire file into a string.

Because the server supports file_get_contents, if the server disables this function, you can use curl. This tool is more powerful than file_get_contents. It is recommended to study "CURL Learning and Application (with Multi-Threading)", you can use curl. Multi-thread download and storage, the effect is even more awesome

Clear file operation cache

clearstatcache() function clears the file status cache. The clearstatcache() function caches return information from certain functions to provide higher performance. But sometimes, such as when you check the same file multiple times in a script and the file is in danger of being deleted or modified during the execution of the script, you need to clear the file status cache in order to get the correct results. To do this, use the clearstatcache() function. Official manual:

Program execution time calculation

Copy code The code is as follows:

/* *

* Get the current time

*/

function getMicrotime() {
list ($usec, $sec) = explode(" ", microtime());
return ((float) $usec + (float) $sec);
}

You can refer to this blog article; "Get PHP page execution time, database read and write times, function call times, etc. [THINKPHP]"

Finally take a look at the effect;



Collected 214 pictures in 409 seconds, downloaded and saved one picture in about 2 seconds, the total size of the picture is about 62M, so It seems:

60*60 can download about 1800 pictures of beautiful women in one hour.

http://www.bkjia.com/PHPjc/327182.htmlwww.bkjia.comtruehttp: //www.bkjia.com/PHPjc/327182.htmlTechArticleThe design idea considers that simply collecting pictures from a web page is too troublesome, so we directly collect his list page to get The URLs of the list are then collected one by one, but php is used to match the list page...