本篇文章主要介绍了PHP实现小偷程序实例,实现了抓取网页咨询和商品信息的功能,具有一定的参考价值,感兴趣的小伙伴们可以参考一下。
为什么使用“小偷程序”?
远程抓取文章资讯或商品信息是很多企业要求程序员实现的功能,也就是俗说的小偷程序。其最主要的优点是:解决了公司网编繁重的工作,大大提高了效率。只需要一运行就能快速的抓取别人网站的信息。
“小偷程序”在哪里运行?
“小偷程序” 应该在 Windows 下的 DOS或 Linux 下通过 PHP 命令运行为最佳,因为,网页运行会超时。
比如图(Windows 下 DOS 为例):
“小偷程序”的实现
这里主要通过一个实例来讲解,我们来抓取下“华强电子网”的资讯信息,请先看观察这个链接 http://www.hqew.com/info-c10.html,当您打开这个页面的时候发现这个页面会发现一些现象:
1、资讯列表有 500 页(2012-01-03);
2、每页的 url 链接都有规律,比如:第1页为http://www.hqew.com/info-c10-1.html;第2页为http://www.hqew.com/info-c10-2.html;……第500页为http://www.hqew.com/info-c10-500.html;
3、由第二点就可以知道,“华强电子网” 的资讯是伪静态或者是生成的静态页面
其实,基本上大部分的网站都有这样的规律,比如:中关村在线、慧聪网、新浪、淘宝……。
这样,我们可以通过这样的思路来实现页面内容的抓取:
1、先获取文章列表页内容;
2、根据文章列表页内容循环获取文章的 url 地址;
3、根据文章的 url 地址获取文章的详细内容
这里,我们主要抓取资讯页里面的:标题(title)、发布如期(date)、作者(author)、来源(source)、内容(content)
“华强电子网”资讯抓取
首先,先建数据表结构,如下所示:
CREATE TABLE `article`.`article` ( `id` MEDIUMINT( 8 ) UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY , `title` VARCHAR( 255 ) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL , `date` VARCHAR( 50 ) NOT NULL , `author` VARCHAR( 100 ) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL , `source` VARCHAR( 100 ) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL , `content` TEXT NOT NULL ) ENGINE = MYISAM CHARACTER SET utf8 COLLATE utf8_general_ci;
抓取程序:
<?php /** * 抓取“华强电子网”资讯程序 * author Lee. * Last modify $Date: 2012-1-3 15:39:35 $ */ header('Content-Type:text/html;Charset=utf-8'); $mysqli = new mysqli('localhost', 'root', '1715544', 'article'); # 数据库连接,请手动修改您自己的数据库信息 $mysqli->set_charset('UTF8'); # 设置数据库编码 function data($url) { global $mysqli; $result = file_get_contents($url); # $result 获取 url 链接内容(注意:这里是文章列表链接) $pattern = '/<li><span class="box_r">.+<\/span><a href="([^"]+)" title=".+" >.+<\/a><\/li>/Usi'; # 取得文章 url 的匹配正则 preg_match_all($pattern, $result, $arr); # 把文章列表 url 分配给数组$arr(二维数组) foreach ($arr[1] as $val) { $val = 'http://www.hqew.com' . $val; # 真实文章 url 地址 $re = file_get_contents($val); # $re 为文章 url 的内容 $pa = '/<p id="article">\s+<h1>(.+)<\/h1>\s+<p id="article\_extinfo">\s+发布:\s+(.+)\s+\|\s+作者:\s+(.+)\s+\|\s+来源:\s+(.*?)\s+<span style="display:none" >.+<p id="article_body">\s*(.+)\s+<\/p>\s+<\/p><!--article end-->/Usi'; # 取得文章内容的正则 preg_match_all($pa, $re, $array); # 把取到的内容分配到数组 $array $content = trim($array[5][0]); $con = array( 'title'=>mysqlString($array[1][0]), 'date'=>mysqlString($array[2][0]), 'author'=>mysqlString(stripAuthorTag($array[3][0])), 'source'=>mysqlString($array[4][0]), 'content'=>mysqlString(stripContentTag($content)) ); $sql = "INSERT INTO article(title,date,author,source,content) VALUES ('{$con['title']}','{$con['date']}','{$con['author']}','{$con['source']}','{$con['content']}')"; $row = $mysqli->query($sql); # 添加到数据库 if ($row) { echo 'add success!'; } else { echo 'add failed!'; } } } /** * stripOfficeTag($v) 对文章内容进行过滤,比如:去掉文章中的链接,过滤掉没用的 HTML 标签…… * @param string $v * @return string */ function stripContentTag($v){ $v = str_replace('<p> </p>', '', $v); $v = str_replace('<p />', '', $v); $v = preg_replace('/<a href=".+" target="\_blank"><strong>(.+)<\/strong><\/a>/Usi', '\1', $v); $v = preg_replace('%(<span\s*[^>]*>(.*)</span>)%Usi', '\2', $v); $v = preg_replace('%(\s+class="Mso[^"]+")%si', '', $v); $v = preg_replace('%( style="[^"]*mso[^>]*)%si', '', $v); $v = preg_replace('/<b><\/b>/', '', $v); return $v; } /** * stripTitleTag($title) 对文章标题进行过滤 * @param string $v * @return string */ function stripAuthorTag($v) { $v = preg_replace('/<a href=".+" target="\_blank">(.+)<\/a>/Usi', '\1', $v); return $v; } /** * mysqlString($str) 过滤数据 * @param string $str * @return string */ function mysqlString($str) { return addslashes(trim($str)); } /** * init($min, $max) 入口程序方法,从 $min 页开始取,到 $max 页结束 * @param int $min 从 1 开始 * @param int $max * @return string 返回 URL 地址 */ function init($min=1, $max) { for ($i=$min; $i<=$max; $i++) { data("http://www.hqew.com/info-c10-{$i}.html"); } } init(1, 500); // 程序入口,从第一页开始抓,抓取500页 ?>
通过上面的程序,就可以实现抓取华强电子网的资讯信息。
入口方法 init($min, $max) 如果想抓取 1-500 页面内容,那么 init(1, 500) 即可!这样,用不了多长时间,华强电子网的资讯就会全部抓取到数据库里面了。^_^
执行界面:
数据库:
以上就是本文的全部内容,希望对大家的学习有所帮助。
相关推荐:
The above is the detailed content of PHP implementation of thief program example. For more information, please follow other related articles on the PHP Chinese website!

PHP type prompts to improve code quality and readability. 1) Scalar type tips: Since PHP7.0, basic data types are allowed to be specified in function parameters, such as int, float, etc. 2) Return type prompt: Ensure the consistency of the function return value type. 3) Union type prompt: Since PHP8.0, multiple types are allowed to be specified in function parameters or return values. 4) Nullable type prompt: Allows to include null values and handle functions that may return null values.

In PHP, use the clone keyword to create a copy of the object and customize the cloning behavior through the \_\_clone magic method. 1. Use the clone keyword to make a shallow copy, cloning the object's properties but not the object's properties. 2. The \_\_clone method can deeply copy nested objects to avoid shallow copying problems. 3. Pay attention to avoid circular references and performance problems in cloning, and optimize cloning operations to improve efficiency.

PHP is suitable for web development and content management systems, and Python is suitable for data science, machine learning and automation scripts. 1.PHP performs well in building fast and scalable websites and applications and is commonly used in CMS such as WordPress. 2. Python has performed outstandingly in the fields of data science and machine learning, with rich libraries such as NumPy and TensorFlow.

Key players in HTTP cache headers include Cache-Control, ETag, and Last-Modified. 1.Cache-Control is used to control caching policies. Example: Cache-Control:max-age=3600,public. 2. ETag verifies resource changes through unique identifiers, example: ETag: "686897696a7c876b7e". 3.Last-Modified indicates the resource's last modification time, example: Last-Modified:Wed,21Oct201507:28:00GMT.

In PHP, password_hash and password_verify functions should be used to implement secure password hashing, and MD5 or SHA1 should not be used. 1) password_hash generates a hash containing salt values to enhance security. 2) Password_verify verify password and ensure security by comparing hash values. 3) MD5 and SHA1 are vulnerable and lack salt values, and are not suitable for modern password security.

PHP is a server-side scripting language used for dynamic web development and server-side applications. 1.PHP is an interpreted language that does not require compilation and is suitable for rapid development. 2. PHP code is embedded in HTML, making it easy to develop web pages. 3. PHP processes server-side logic, generates HTML output, and supports user interaction and data processing. 4. PHP can interact with the database, process form submission, and execute server-side tasks.

PHP has shaped the network over the past few decades and will continue to play an important role in web development. 1) PHP originated in 1994 and has become the first choice for developers due to its ease of use and seamless integration with MySQL. 2) Its core functions include generating dynamic content and integrating with the database, allowing the website to be updated in real time and displayed in personalized manner. 3) The wide application and ecosystem of PHP have driven its long-term impact, but it also faces version updates and security challenges. 4) Performance improvements in recent years, such as the release of PHP7, enable it to compete with modern languages. 5) In the future, PHP needs to deal with new challenges such as containerization and microservices, but its flexibility and active community make it adaptable.

The core benefits of PHP include ease of learning, strong web development support, rich libraries and frameworks, high performance and scalability, cross-platform compatibility, and cost-effectiveness. 1) Easy to learn and use, suitable for beginners; 2) Good integration with web servers and supports multiple databases; 3) Have powerful frameworks such as Laravel; 4) High performance can be achieved through optimization; 5) Support multiple operating systems; 6) Open source to reduce development costs.


Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

AI Hentai Generator
Generate AI Hentai for free.

Hot Article

Hot Tools

SecLists
SecLists is the ultimate security tester's companion. It is a collection of various types of lists that are frequently used during security assessments, all in one place. SecLists helps make security testing more efficient and productive by conveniently providing all the lists a security tester might need. List types include usernames, passwords, URLs, fuzzing payloads, sensitive data patterns, web shells, and more. The tester can simply pull this repository onto a new test machine and he will have access to every type of list he needs.

Safe Exam Browser
Safe Exam Browser is a secure browser environment for taking online exams securely. This software turns any computer into a secure workstation. It controls access to any utility and prevents students from using unauthorized resources.

Atom editor mac version download
The most popular open source editor

MinGW - Minimalist GNU for Windows
This project is in the process of being migrated to osdn.net/projects/mingw, you can continue to follow us there. MinGW: A native Windows port of the GNU Compiler Collection (GCC), freely distributable import libraries and header files for building native Windows applications; includes extensions to the MSVC runtime to support C99 functionality. All MinGW software can run on 64-bit Windows platforms.

Dreamweaver CS6
Visual web development tools