search
Homephp教程php手册网页抓取:PHP实现网页爬虫方式小结

网页抓取:PHP实现网页爬虫方式小结

Jun 06, 2016 pm 07:47 PM
httpphpaccomplishcrawlWaysourcereptileWeb page

来源:http://www.ido321.com/1158.html 抓取某一个网页中的内容,需要对DOM树进行解析,找到指定节点后,再抓取我们需要的内容,过程有点繁琐。LZ总结了几种常用的、易于实现的网页抓取方式,如果熟悉JQuery选择器,这几种框架会相当简单。 一、Ganon 项目

来源:http://www.ido321.com/1158.html


网页抓取:PHP实现网页爬虫方式小结

抓取某一个网页中的内容,需要对DOM树进行解析,找到指定节点后,再抓取我们需要的内容,过程有点繁琐。LZ总结了几种常用的、易于实现的网页抓取方式,如果熟悉JQuery选择器,这几种框架会相当简单。

一、Ganon

项目地址: http://code.google.com/p/ganon/

文档: http://code.google.com/p/ganon/w/list

测试:抓取我的网站首页所有class属性值是focus的div元素,并且输出class值

<span><?php <span>include</span> <span>'ganon.php'</span>;
 $html = file_get_dom(<span>'http://www.ido321.com/'</span>);
 <span>foreach</span>($html(<span>'div[class="focus"]'</span>) <span>as</span> $element) {
   <span>echo</span> $element-><span>class</span>, <span>"<br>\n"</span>; 
 }
?>

 

结果:

网页抓取:PHP实现网页爬虫方式小结

二、phpQuery

项目地址:http://code.google.com/p/phpquery/

文档:https://code.google.com/p/phpquery/wiki/Manual

<span><?php <span>include</span> <span>'phpQuery/phpQuery.php'</span>; 
phpQuery::newDocumentFile(<span>'http://www.ido321.com/'</span>); 
$artlist = pq(<span>"article"</span>); 
<span>foreach</span>($artlist <span>as</span> $title){ 
   <span>echo</span> pq($title)->find(<span>'h2'</span>)->html().<span>"<br>"</span>; 
} 
?>

 

结果:

网页抓取:PHP实现网页爬虫方式小结

  三、Simple-Html-Dom

项目地址: http://simplehtmldom.sourceforge.net/
文档: http://simplehtmldom.sourceforge.net/manual.htm

测试:抓取我网站首页的所有链接

<span><?php <span>include</span> <span>'simple_html_dom.php'</span>;
<span>//使用url和file都可以创建DOM</span>
$html = file_get_html(<span>'http://www.ido321.com/'</span>);

<span>//找到所有图片</span>
<span>// foreach($html->find('img') as $element)</span>
<span>//        echo $element->src . '<br>';</span>

<span>//找到所有链接</span>
<span>foreach</span>($html->find(<span>'a'</span>) <span>as</span> $element)
       <span>echo</span> $element->href . <span>'<br>'</span>; 
?>

 

结果:(截图是一部分)

网页抓取:PHP实现网页爬虫方式小结

四、Snoopy

项目地址:http://code.google.com/p/phpquery/

文档:http://code.google.com/p/phpquery/wiki/Manual

测试:抓取我的网站首页

<span><?php <span>include</span>(<span>"Snoopy.class.php"</span>);
$url = <span>"http://www.ido321.com"</span>;
$snoopy = <span>new</span> Snoopy;
$snoopy->fetch($url); <span>//获取所有内容</span>
 <span>echo</span> $snoopy->results; <span>//显示结果</span>
<span>// echo $snoopy->fetchtext ;//获取文本内容(去掉html代码)</span>
<span>// echo $snoopy->fetchlinks($url) ;//获取链接</span>
<span>// $snoopy->fetchform ;//获取表单 </span>
?>

 

结果:

网页抓取:PHP实现网页爬虫方式小结

五、手动编写爬虫

如果编写能力ok,可以手写一个网页爬虫,实现网页抓取。网上有千篇一律的介绍此方法的文章,LZ就不赘述了。有兴趣了解的,可以百度  php 网页抓取。

ps:资源分享

常见的开源爬虫项目请戳:http://blog.chinaunix.net/uid-22414998-id-3774291.html


国民岳父的“屁民理论”



Statement
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

Video Face Swap

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Tools

MantisBT

MantisBT

Mantis is an easy-to-deploy web-based defect tracking tool designed to aid in product defect tracking. It requires PHP, MySQL and a web server. Check out our demo and hosting services.

mPDF

mPDF

mPDF is a PHP library that can generate PDF files from UTF-8 encoded HTML. The original author, Ian Back, wrote mPDF to output PDF files "on the fly" from his website and handle different languages. It is slower than original scripts like HTML2FPDF and produces larger files when using Unicode fonts, but supports CSS styles etc. and has a lot of enhancements. Supports almost all languages, including RTL (Arabic and Hebrew) and CJK (Chinese, Japanese and Korean). Supports nested block-level elements (such as P, DIV),

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

DVWA

DVWA

Damn Vulnerable Web App (DVWA) is a PHP/MySQL web application that is very vulnerable. Its main goals are to be an aid for security professionals to test their skills and tools in a legal environment, to help web developers better understand the process of securing web applications, and to help teachers/students teach/learn in a classroom environment Web application security. The goal of DVWA is to practice some of the most common web vulnerabilities through a simple and straightforward interface, with varying degrees of difficulty. Please note that this software

ZendStudio 13.5.1 Mac

ZendStudio 13.5.1 Mac

Powerful PHP integrated development environment