Introduction to PHP crawling and collection class snoopy_PHP tutorial

Snoopy is a php class that is used to imitate the functions of a web browser. It can complete the tasks of obtaining web content and sending forms. Official website http://snoopy.sourceforge.net/

Some features of Snoopy:

• Fetch the content of the web page fetch()
• Fetch the text content of the web page (remove HTML tags) fetchtext()
• Catch webpage links, form fetchlinks() fetchform()
• Support proxy host
• Supports basic username/password verification
• Support setting user_agent, referer (source), cookies and header content (header file)
• Supports browser redirection and can control redirection depth
• Can expand links in web pages into high-quality URLs (default)
• Submit data and get the return value
• Support tracking HTML framework
• Supports passing cookies when redirecting

Requires php4 or above. Since it is a PHP class, there is no need to expand support. It is the best choice when the server does not support curl.

Class method

1. fetch($uri)

This is the method used to crawl the content of web pages. The $URI parameter is the URL address of the crawled web page. The fetched results are stored in $this->results.

If you are scraping a frame, Snoopy will track each frame and store it in an array, and then store it in $this->results.

<?php
$url = "http://www.bkjia.com/librarys/veda/";
include("./Snoopy.class.php");

$snoopy = new Snoopy;
$snoopy->fetch($url); 		//获取所有内容
echo $snoopy->results; 		//显示结果
?>

2. fetchtext($URI)

This method is similar to fetch(). The only difference is that this method will remove HTML tags and other irrelevant data and only return the text content in the web page.

<?php
$url = "http://www.bkjia.com/librarys/veda/";
include("./Snoopy.class.php");

$snoopy = new Snoopy;
$snoopy->fetchtext($url); 		//获取文本内容
echo $snoopy->results; 		//显示结果
?>

3. fetchform($URI)

This method is similar to fetch(). The only difference is that this method will remove HTML tags and other irrelevant data, and only return the form content (form) in the web page.

4. fetchlinks($URI)

This method is similar to fetch(). The only difference is that this method will remove HTML tags and other irrelevant data, and only return links in the web page. By default, relative links will be automatically completed and converted into full URLs.

5. submit($URI,$formvars)

This method sends a confirmation form to the link address specified by $URL. $formvars is an array that stores form parameters.

6. submittext($URI,$formvars)

This method is similar to submit(). The only difference is that this method will remove HTML tags and other irrelevant data, and only return the text content of the web page after login.

7. submitlinks($URI)

This method is similar to submit(). The only difference is that this method will remove HTML tags and other irrelevant data, and only return the link in the web page. By default, relative links will be automatically completed and converted into full URLs.

Class attributes (default values ​​are in brackets)

• $host The connected host
• $port The port to connect to
• $proxy_host The proxy host to use, if any
• $proxy_port The proxy host port to use, if any
• $agent User Agent Disguise (Snoopy v0.1)
• $referer source information, if available
• $cookies cookies, if any
• $rawheaders Other header information, if any
• $maxredirs maximum number of redirects, 0=not allowed (5)
• $offsiteok whether or not to allow redirects off-site. (true)
• $expandlinks Whether to complete all links to complete addresses (true)
• $user authentication username, if available
• $pass authentication username, if available
• $accept http accept type (image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, */*)
• $error where the error was reported, if any
• $response_code The response code returned from the server
• $headers header information returned from the server
• $maxlength is the longest returned data length
• $read_timeout read operation timeout (requires PHP 4 Beta 4+), set to 0 for no timeout
• $timed_out If a read operation times out, this attribute returns true (requires PHP 4 Beta 4+)
• $maxframes is the maximum number of frames allowed to be tracked
• $status The status of the captured http
• $temp_dir The temporary file directory (/tmp) that the web server can write to
• $curl_path cURL binary directory, if there is no cURL binary, set it to false

Demo

 include "Snoopy.class.php";
 $snoopy = new Snoopy;
 
 $snoopy->proxy_host = "http://www.bkjia.com/librarys/veda/";
 $snoopy->proxy_port = "80";
 
 $snoopy->agent = "(compatible; MSIE 4.01; MSN 2.5; AOL 4.0; Windows 98)";
 $snoopy->referer = "http://www.4wei.cn";
 
 $snoopy->cookies["SessionID"] = 238472834723489l;
 $snoopy->cookies["favoriteColor"] = "RED";
 
 $snoopy->rawheaders["Pragma"] = "no-cache";
 
 $snoopy->maxredirs = 2;
 $snoopy->offsiteok = false;
 $snoopy->expandlinks = false;
 
 $snoopy->user = "joe";
 $snoopy->pass = "bloe";
 
 if($snoopy->fetchtext("http://www.4wei.cn"))
 {
 echo "<PRE>".htmlspecialchars($snoopy->results)."

n"; } else echo "error fetching document: ".$snoopy->error."n";

Get the specified url content:

 <?
 $url = "http://www.bkjia.com/librarys/veda/";
 include("snoopy.php");
 $snoopy = new Snoopy;
 $snoopy->fetch($url); //获取所有内容
 echo $snoopy->results; //显示结果
 //可选以下
 //$snoopy->fetchtext //获取文本内容（去掉html代码）
 //$snoopy->fetchlinks //获取链接
 //$snoopy->fetchform  //获取表单
 ?>

Form submission:

<?php
$formvars["username"] = "admin";
$formvars["pwd"] = "admin";
$action = "http://www.bkjia.com/librarys/veda/";//表单提交地址
$snoopy->submit($action,$formvars);//$formvars为提交的数组
echo $snoopy->results; //获取表单提交后的 返回的结果
//可选以下
$snoopy->submittext; //提交后只返回 去除html的 文本
$snoopy->submitlinks;//提交后只返回 链接
?>

Now that the form has been submitted, there are a lot of things you can do. Next, let’s disguise the IP and browser:

<?php
$formvars["username"] = "admin";
$formvars["pwd"] = "admin";
$action = "http://www.4wei.cn";
include "snoopy.php";
$snoopy = new Snoopy;
$snoopy->cookies["PHPSESSID"] = 'fc106b1918bd522cc863f36890e6fff7'; //伪装sessionid
$snoopy->agent = "(compatible; MSIE 4.01; MSN 2.5; AOL 4.0; Windows 98)"; //伪装浏览器
$snoopy->referer = http://www.4wei.cn; //伪装来源页地址 http_referer
$snoopy->rawheaders["Pragma"] = "no-cache"; //cache 的http头信息
$snoopy->rawheaders["X_FORWARDED_FOR"] = "127.0.0.101"; //伪装ip
$snoopy->submit($action,$formvars);
echo $snoopy->results;
?>

It turns out that we can camouflage session, camouflage browser, camouflage IP, haha ​​we can do a lot of things. For example, if you bring a verification code and verify your IP to vote, you can vote non-stop.

ps: Disguising the IP here is actually disguising the http header, so the IP obtained through REMOTE_ADDR cannot be disguised. On the contrary, those who obtain the IP through the http header (the kind that can prevent proxying) can create their own IP. .

关于如何验证码 ，简单说下：首先用普通的浏览器， 查看页面 ， 找到验证码所对应的sessionid，同时记下sessionid和验证码值，接下来就用snoopy去伪造 。

原理：由于是同一个sessionid 所以取得的验证码和第一次输入的是一样的。

有时我们可能需要伪造更多的东西，snoopy完全为我们想到了：

<?php
$snoopy->proxy_host = "http://www.bkjia.com/librarys/veda/";
$snoopy->proxy_port = "8080"; //使用代理
$snoopy->maxredirs = 2; //重定向次数
$snoopy->expandlinks = true; //是否补全链接 在采集的时候经常用到
// 例如链接为 /images/taoav.gif 可改为它的全链接 http://www.4wei.cn/images/taoav.gif
$snoopy->maxframes = 5 //允许的最大框架数
//注意抓取框架的时候 $snoopy->results 返回的是一个数组
$snoopy->error //返回报错信息
?>

比较完整的示例：

/**
* You need the snoopy.class.php from 
* http://snoopy.sourceforge.net/
*/
include("snoopy.class.php");
 
$snoopy = new Snoopy;
// need an proxy?:
//$snoopy->proxy_host = "my.proxy.host";
//$snoopy->proxy_port = "8080";
 
// set browser and referer:
$snoopy->agent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)";
$snoopy->referer = "http://www.jonasjohn.de/";
 
// set some cookies:
$snoopy->cookies["SessionID"] = '238472834723489';
$snoopy->cookies["favoriteColor"] = "blue";
 
// set an raw-header:
$snoopy->rawheaders["Pragma"] = "no-cache";
 
// set some internal variables:
$snoopy->maxredirs = 2;
$snoopy->offsiteok = false;
$snoopy->expandlinks = false;
 
// set username and password (optional)
//$snoopy->user = "joe";
//$snoopy->pass = "bloe";
 
// fetch the text of the website www.google.com:
if($snoopy->fetchtext("http://www.google.com")){ 
    // other methods: fetch, fetchform, fetchlinks, submittext and submitlinks
 
    // response code:
    print "response code: ".$snoopy->response_code."<br/>n";
 
    // print the headers:
 
    print "<b>Headers:</b><br/>";
    while(list($key,$val) = each($snoopy->headers)){
        print $key.": ".$val."<br/>n";
    }
 
    print "<br/>n";
 
    // print the texts of the website:
    print htmlspecialchars($snoopy->results)."n";
}
else {
    print "Snoopy: error while fetching document: ".$snoopy->error."n";
}

用Snoopy类完成一个简单的图片采集：

<meta http-equiv='content-type' content='text/html;charset=utf-8'>
<?php    
include 'Snoopy.class.php';   //加载Snoopy类     
$snoopy = new Snoopy();       //实例化一个对象
$sourceURL = "http://www.bkjia.com/librarys/veda/";    //要抓取的网页
$snoopy->fetchlinks($sourceURL);        //获得网页的链接
$a = $snoopy->results;     //得到网页链接的结果
$re = "/d+.html$/";     //匹配的正则
//过滤获取指定的文件地址请求  
foreach ($a as $tmp) { 
	if (preg_match($re, $tmp)) {
		$aa=$tmp;        
 	}    
}  
getImgURL($aa);
function getImgURL($siteName) 
{        
 	$snoopy = new Snoopy();        
 	$snoopy->fetch($siteName);                
 	$fileContent = $snoopy->results;    //获取过滤后的页面的内容            
 	//匹配图片的正则表达式        
 	$reTag = "/<img[^s]+src="(http://[^"]+).(jpg|png|gif|jpeg)"[^/]*/>/i";                
	if (preg_match($reTag, $fileContent)) {  
   		//过滤图片
		$ret = preg_match_all($reTag, $fileContent, $matchResult);                     
		for ($i = 0, $len = count($matchResult[1]); $i < $len; ++$i) 
		{      
			saveImgURL($matchResult[1][$i], $matchResult[2][$i]);            
		}        
	}    
}        
function saveImgURL($name, $suffix) { 
	$url = $name.".".$suffix;                
 	echo "请求的图片地址：".$url."<br/>";                
 	$imgSavePath = "E:/123/images/";  //图片保存地址      
 	$imgId =mt_rand(); //产生一个随机的文件名
 	if ($suffix == "gif") { 
		//根据图片类型，放入不同的文件夹下面           
  		$imgSavePath .= "emotion";        
 	} 
	else 
	{            
  		$imgSavePath .= "topic";        
 	}        
 	$imgSavePath .= ("/".$imgId.".".$suffix);  //组装要保存的文件名
 	if (is_file($imgSavePath)) {   
		//判断文件名是否存在，存在则删除         
  		unlink($imgSavePath);            
  		echo "<p style='color:#f00;'>文件".$imgSavePath."已存在，将被删除</p>";        
 	}  
 	$imgFile = file_get_contents($url); //读取网络文件     
 	$flag = file_put_contents($imgSavePath,$imgFile);   //写入到本地 
 	if ($flag) {            
  		echo "<p>文件".$imgSavePath."保存成功</p>";        
 	}    
}
?>

http://www.bkjia.com/PHPjc/752536.htmlwww.bkjia.comtruehttp://www.bkjia.com/PHPjc/752536.htmlTechArticlesnoopy是一个php类，用来模仿web浏览器的功能，它能完成获取网页内容和发送表单的任务。官方网站 http://snoopy.sourceforge.net/ Snoopy的一些功能...