search

Home  >  Article  >  Backend Development  >  PHP collection class snoopy example introduction

PHP collection class snoopy example introduction

小云云
小云云Original
2018-03-10 13:11:348425browse

Snoopy is a php class that is used to imitate the functions of a web browser. It can complete the tasks of obtaining web content and sending forms. Official website http://snoopy.sourceforge.net/

Some features of Snoopy:

  • Fetch the content of the web page fetch()

  • Capture the text content of the web page (remove HTML tags) fetchtext()

  • Capture the links of the web page, form fetchlinks() fetchform()

  • Support proxy host

  • Support basic username/password verification

  • Support setting user_agent, referer(origin) ), cookies and header content (header file)

  • Supports browser redirection and can control the redirection depth

  • Can put the web page in The link is expanded into a high-quality URL (default)

  • Submit data and get the return value

  • Support tracking HTML framework

  • Supports passing cookies when redirecting

Requires php4 or above. Since it is a PHP class, there is no need to expand support. It is the best choice when the server does not support curl.

Class method

1. fetch($uri)

This is the method used to grab the content of the web page. The $URI parameter is the URL address of the crawled web page. The fetched results are stored in $this->results.

If you are scraping a frame, Snoopy will track each frame and store it in an array, and then store it in $this->results. 

  1. <?php  
$url = "http://www.nowamagic.net/librarys/veda/";  
include("./Snoopy.class.php");  
$snoopy = new Snoopy;  
$snoopy->fetch($url);        //获取所有内容
echo $snoopy->results;       //显示结果
?>

2. fetchtext($URI)

This method is similar to fetch(), the only difference is that this method will remove HTML tags And other irrelevant data, only the text content in the web page is returned. 

  1. <?php  
$url = "http://www.nowamagic.net/librarys/veda/";  
include("./Snoopy.class.php");  
$snoopy = new Snoopy;  
$snoopy->fetchtext($url);        //获取文本内容
echo $snoopy->results;       //显示结果
?>

3. fetchform($URI)

This method is similar to fetch(), the only difference is that this method will remove HTML tags And other irrelevant data, only the form content (form) in the web page is returned.

4. fetchlinks($URI)

This method is similar to fetch(). The only difference is that this method will remove HTML tags and other irrelevant data, and only return the links in the web page (link ). By default, relative links will be automatically completed and converted into full URLs.

5. submit($URI,$formvars)

This method sends the confirmation form to the link address specified by $URL. $formvars is an array that stores form parameters.

6. submittext($URI,$formvars)

This method is similar to submit(). The only difference is that this method will remove HTML tags and other irrelevant data, and only return the login information. Text content in web pages.

7. submitlinks($URI)

This method is similar to submit(). The only difference is that this method will remove HTML tags and other irrelevant data, and only return the links in the web page (link ). By default, relative links will be automatically completed and converted into full URLs.

Class attributes (default values ​​are in brackets)

  • $host The connected host

  • $port The connected port

  • $proxy_host The proxy host to use, if any

  • $proxy_port The proxy host port to use, if any

  • $agent User agent masquerading (Snoopy v0.1)

  • $referer source information, if any

  • $cookies cookies, if any

  • $rawheaders other header information, if any

  • $maxredirs maximum Number of redirects, 0=not allowed (5)

  • $offsiteok whether or not to allow redirects off-site. (true)

  • $expandlinks Whether to complete all links to complete addresses (true)

  • $user authentication username, if any

  • $pass Authentication username, if any

  • $accept http accept type(image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, */*)

  • $error Where is the error reported, if any

  • $response_code The response code returned from the server

  • $headers Header information returned from the server

  • $maxlength The longest returned data length

  • $read_timeout Read operation timeout (requires PHP 4 Beta 4+), set to 0 for no timeout

  • $timed_out If a read operation times out, this attribute returns true (requires PHP 4 Beta 4+)

  • $maxframes The maximum number of frames allowed to be tracked

  • $status The status of the captured http

  • $temp_dir web page The temporary file directory (/tmp) that the server can write to

  • $curl_path cURL binary directory, if there is no cURL binary, set it to false

Demo

  1. include "Snoopy.class.php";  
$snoopy = new Snoopy;  
$snoopy->proxy_host = "http://www.nowamagic.net/librarys/veda/";  
$snoopy->proxy_port = "80";  
$snoopy->agent = "(compatible; MSIE 4.01; MSN 2.5; AOL 4.0; Windows 98)";  
$snoopy->referer = "http://www.4wei.cn";  
$snoopy->cookies["SessionID"] = 238472834723489l;  
$snoopy->cookies["favoriteColor"] = "RED";  
$snoopy->rawheaders["Pragma"] = "no-cache";  
$snoopy->maxredirs = 2;  
$snoopy->offsiteok = false;  
$snoopy->expandlinks = false;  
$snoopy->user = "joe";  
$snoopy->pass = "bloe";  
if($snoopy->fetchtext("http://www.4wei.cn"))  
{  
echo "<PRE>".htmlspecialchars($snoopy->results)."
    n"; } else echo "error fetching document: ".$snoopy->error."n";

Get the content of the specified url:

  1. <?  
$url = "http://www.nowamagic.net/librarys/veda/";  
include("snoopy.php");  
$snoopy = new Snoopy;  
$snoopy->fetch($url); //获取所有内容
echo $snoopy->results; //显示结果
//可选以下
//$snoopy->fetchtext //获取文本内容（去掉html代码）
//$snoopy->fetchlinks //获取链接
//$snoopy->fetchform  //获取表单
?>

Form submission:

  1. <?php  
$formvars["username"] = "admin";  
$formvars["pwd"] = "admin";  
$action = "http://www.nowamagic.net/librarys/veda/";//表单提交地址  
$snoopy->submit($action,$formvars);//$formvars为提交的数组
echo $snoopy->results; //获取表单提交后的 返回的结果
//可选以下
$snoopy->submittext; //提交后只返回 去除html的 文本
$snoopy->submitlinks;//提交后只返回 链接
?>

Now that the form has been submitted, you can do a lot of things. Next, let’s disguise the IP and browser: 

  1. <?php  
$formvars["username"] = "admin";  
$formvars["pwd"] = "admin";  
$action = "http://www.4wei.cn";  
include "snoopy.php";  
$snoopy = new Snoopy;  
$snoopy->cookies["PHPSESSID"] = &#39;fc106b1918bd522cc863f36890e6fff7&#39;; //伪装sessionid
$snoopy->agent = "(compatible; MSIE 4.01; MSN 2.5; AOL 4.0; Windows 98)"; //伪装浏览器
$snoopy->referer = http://www.4wei.cn; //伪装来源页地址 http_referer
$snoopy->rawheaders["Pragma"] = "no-cache"; //cache 的http头信息
$snoopy->rawheaders["X_FORWARDED_FOR"] = "127.0.0.101"; //伪装ip
$snoopy->submit($action,$formvars);  
echo $snoopy->results;

  2. ?>

原来我们可以伪装session 伪装浏览器 ，伪装ip， haha 可以做很多事情了。例如 带验证码，验证ip 投票， 可以不停的投。

ps:这里伪装ip ，其实是伪装http头，所以一般的通过 REMOTE_ADDR 获取的ip是伪装不了，反而那些通过http头来获取ip的(可以防止代理的那种) 就可以自己来制造ip。

关于如何验证码 ，简单说下：首先用普通的浏览器， 查看页面 ， 找到验证码所对应的sessionid，同时记下sessionid和验证码值，接下来就用snoopy去伪造 。

原理：由于是同一个sessionid 所以取得的验证码和第一次输入的是一样的。

有时我们可能需要伪造更多的东西，snoopy完全为我们想到了：

  1. <?php  
$snoopy->proxy_host = "http://www.nowamagic.net/librarys/veda/";  
$snoopy->proxy_port = "8080"; //使用代理
$snoopy->maxredirs = 2; //重定向次数
$snoopy->expandlinks = true; //是否补全链接 在采集的时候经常用到
// 例如链接为 /images/taoav.gif 可改为它的全链接 <a href="http://www.4wei.cn/images/taoav.gif">http://www.4wei.cn/images/taoav.gif</a>
$snoopy->maxframes = 5 //允许的最大框架数
//注意抓取框架的时候 $snoopy->results 返回的是一个数组
$snoopy->error //返回报错信息
?>

比较完整的示例：

  1. /**
* You need the snoopy.class.php from 
* http://snoopy.sourceforge.net/
*/
include("snoopy.class.php");  
$snoopy = new Snoopy;  
// need an proxy?:
//$snoopy->proxy_host = "my.proxy.host";
//$snoopy->proxy_port = "8080";
// set browser and referer:
$snoopy->agent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)";  
$snoopy->referer = "http://www.jonasjohn.de/";  
// set some cookies:
$snoopy->cookies["SessionID"] = &#39;238472834723489&#39;;  
$snoopy->cookies["favoriteColor"] = "blue";  
// set an raw-header:
$snoopy->rawheaders["Pragma"] = "no-cache";  
// set some internal variables:
$snoopy->maxredirs = 2;  
$snoopy->offsiteok = false;  
$snoopy->expandlinks = false;  
// set username and password (optional)
//$snoopy->user = "joe";
//$snoopy->pass = "bloe";
// fetch the text of the website www.google.com:
if($snoopy->fetchtext("http://www.google.com")){   
    // other methods: fetch, fetchform, fetchlinks, submittext and submitlinks
    // response code:
    print "response code: ".$snoopy->response_code."<br/>n";  
    // print the headers:
    print "<b>Headers:</b><br/>";  
    while(list($key,$val) = each($snoopy->headers)){  
        print $key.": ".$val."<br/>n";  
    }  
    print "<br/>n";  
    // print the texts of the website:
    print htmlspecialchars($snoopy->results)."n";  
}  
else {  
    print "Snoopy: error while fetching document: ".$snoopy->error."n";  
}

用Snoopy类完成一个简单的图片采集：

  1. <meta http-equiv=&#39;content-type&#39; content=&#39;text/html;charset=utf-8&#39;>  
<?php      
include &#39;Snoopy.class.php&#39;;   //加载Snoopy类     
$snoopy = new Snoopy();       //实例化一个对象
$sourceURL = "http://www.nowamagic.net/librarys/veda/";    //要抓取的网页  
$snoopy->fetchlinks($sourceURL);        //获得网页的链接
$a = $snoopy->results;     //得到网页链接的结果
$re = "/d+.html$/";     //匹配的正则
//过滤获取指定的文件地址请求  
foreach ($a as $tmp) {   
    if (preg_match($re, $tmp)) {  
        $aa=$tmp;          
    }      
}    
getImgURL($aa);  
function getImgURL($siteName)   
{          
    $snoopy = new Snoopy();          
    $snoopy->fetch($siteName);                  
    $fileContent = $snoopy->results;    //获取过滤后的页面的内容            
    //匹配图片的正则表达式        
    $reTag = "/<img[^s]+src="(http://[^"]+).(jpg|png|gif|jpeg)"[^/]*/>/i";                
    if (preg_match($reTag, $fileContent)) {    
        //过滤图片
        $ret = preg_match_all($reTag, $fileContent, $matchResult);                       
        for ($i = 0, $len = count($matchResult[1]); $i < $len; ++$i)   
        {        
            saveImgURL($matchResult[1][$i], $matchResult[2][$i]);              
        }          
    }      
}          
function saveImgURL($name, $suffix) {   
    $url = $name.".".$suffix;                  
    echo "请求的图片地址：".$url."<br/>";                  
    $imgSavePath = "E:/123/images/";  //图片保存地址      
    $imgId =mt_rand(); //产生一个随机的文件名
    if ($suffix == "gif") {   
        //根据图片类型，放入不同的文件夹下面           
        $imgSavePath .= "emotion";          
    }   
    else
    {              
        $imgSavePath .= "topic";          
    }          
    $imgSavePath .= ("/".$imgId.".".$suffix);  //组装要保存的文件名
    if (is_file($imgSavePath)) {     
        //判断文件名是否存在，存在则删除         
        unlink($imgSavePath);              
        echo "<p style=&#39;color:#f00;&#39;>文件".$imgSavePath."已存在，将被删除</p>";          
    }    
    $imgFile = file_get_contents($url); //读取网络文件     
    $flag = file_put_contents($imgSavePath,$imgFile);   //写入到本地 
    if ($flag) {              
        echo "<p>文件".$imgSavePath."保存成功</p>";          
    }      
}  
?>

相关推荐：

php使用snoopy与curl模拟登陆的实例分享

php数据抓取类Snoopy使用

snoopy（强大的PHP采集类） 详细介绍

The above is the detailed content of PHP collection class snoopy example introduction. For more information, please follow other related articles on the PHP Chinese website!

Statement：
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn
Previous article：How to operate QQ third-party login in PHPNext article：How to operate QQ third-party login in PHP

Related articles

See more