How to use PHP and phpSpider to accurately crawl specific website content?

How to use PHP and phpSpider to accurately crawl specific website content?

Introduction:
With the development of the Internet, the amount of data on the website is increasing, and it is inefficient to obtain the required information through manual operations. Therefore, we often need to use automated crawling tools to obtain the content of specific websites. The PHP language and phpSpider library are one of the very practical tools. This article will introduce how to use PHP and phpSpider to accurately crawl specific website content, and provide code examples.

1. Install phpSpider

First, we need to install the phpSpider library in the local environment. We can install it through Composer, open the terminal, enter the project directory, and then execute the following command:

composer require phpspider/phpspider

After executing this command, phpSpider will be installed to our project in the directory.

2. Create a crawling script

Next, we need to create a PHP script to crawl website content. We can use IDE tools (such as Sublime Text, PHPStorm, etc.) to open a blank PHP file and start writing code.

The following is a simple sample code for crawling news titles and content on a specified website:

require 'vendor/autoload.php ';

use phpspidercorephpspider;
use phpspidercoreequests;
use phpspidercoreselector;

// Set encoding
header("Content-type:text/html;charset=utf -8");

// Set the target website for crawling
$url = "http://www.example.com/news";

// Set proxy
requests::set_proxy(['127.0.0.1:8888']);

// Set user agent
requests::set_useragent(

'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:7.0) Gecko/20100101 Firefox/9.0.1'

);

// Define the callback function to process each page
function handle_page($page)
{

$selector = "//div[@class='news-item']";
$result = selector::select($page['raw'], $selector);

foreach ($result as $item) {
    $title = selector::select($item, "//h2/a/text()");
    $content = selector::select($item, "//div[@class='content']/text()");
    
    // 输出抓取的结果
    echo "标题: ".$title.PHP_EOL;
    echo "内容: ".$content.PHP_EOL;
    echo "-----------------------------".PHP_EOL;
}

}

// Create a crawling instance
$spider = new phpspider();

// Set the crawled URL
$spider->add_url($url);

// Set the callback function after the page crawling is completed
$spider->on_scan_page = function ($page, $content, $phpspider) {

handle_page($page);

};

// Start crawling
$spider-> start();

?>

Note: "http://www.example.com/news" in the above code is an example link. Please replace it with yours when using it. The website link to crawl.

3. Code Analysis

In the above code, we first import the phpspider library, then set the target website URL to be crawled, and set related configurations such as proxy and user agent. Next, we define a callback function handle_page to process each page. In this callback function, we use the selector class provided by phpSpider to parse the page and extract the required news titles and content. Finally, we output the crawl results.

Next, we created a phpspider instance, added the URL to be crawled and set the on_scan_page callback function, and then started the crawling process.

4. Summary

By using PHP and phpSpider, we can easily achieve precise crawling of specific website content. You only need to install the phpSpider library, write a crawl script and configure relevant parameters to automatically obtain the required data. I hope this article can help you learn and understand how to use PHP and phpSpider to crawl website content.

References:

1. phpSpider official documentation: http://phpspider.org/
2. Composer official website: https://getcomposer.org/

The above is the detailed content of How to use PHP and phpSpider to accurately crawl specific website content?. For more information, please follow other related articles on the PHP Chinese website!