Basic crawler tutorial: PHP program implements single-threaded and multi-threaded data crawling

Basic crawler tutorial: PHP program implements single-threaded and multi-threaded data capture

With the development of Internet technology, crawler technology is increasingly widely used in various data acquisition scenarios. Simply put, crawler technology simulates browser behavior, initiates requests to target websites, obtains web content, and performs data extraction and analysis.

In PHP programs, crawler technology can be implemented with the help of third-party components, such as cURL and Simple HTML DOM. These components greatly simplify the workload of data crawling. This article will introduce how to use PHP to implement single-threaded and multi-threaded data capture.

1. Single-threaded data crawling

Single-threaded data crawling simply means crawling each page of the target site sequentially in one thread, and then extracting and parsing data on the page. .

In PHP, you can use the cURL component to obtain single-page data. Here is a simple example.

<?php

// 进行 cURL 初始化
$ch = curl_init();

// 设置 URL 和其他参数
curl_setopt($ch, CURLOPT_URL, "http://example.com/");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HEADER, 0);

// 获取页面内容
$page_data = curl_exec($ch);

// 关闭 cURL 
curl_close($ch);

?>

Through the above code, you can get the homepage content of the specified website. Next, you can use the Simple HTML DOM component to analyze the page content and obtain the required data.

<?php
// 引入 Simple HTML DOM 组件
require_once('simple_html_dom.php');

// 获取页面内容
$page_data = file_get_html('http://example.com/');

// 获取某个节点
$node = $page_data->find('#content', 0);

// 获取节点文本
$text = $node->plaintext;

?>

Through the above code, you can obtain the text content of the node with the id of content on the home page of the specified website.

2. Multi-threaded data capture

The limitation of single-threaded data capture is its slow speed, but it can meet the needs in simple data acquisition scenarios, but if you need to obtain more If you have a lot of data, you need to consider using multi-threading to capture it.

The principle of multi-threaded data capture is to divide data request and data processing into multiple threads and perform them simultaneously, which can improve the efficiency of the entire data acquisition and analysis. In PHP, you can use multiple methods to implement multi-threaded data capture, such as using multiple curl components, using multi-process methods, etc. Here is an example of using curl multi-threading.

<?php 

// 定义目标 URL 列表
$urls = array(
    'http://example.com/1.html',
    'http://example.com/2.html',
    'http://example.com/3.html',
    'http://example.com/4.html',
    'http://example.com/5.html'
);

// 线程数
$max_threads = 5;

$curlopt_array = array();

// 创建多个 cURL 组件
for ($i = 0; $i < $max_threads; $i++) {
    $ch[$i] = curl_init();
}

// 设置 cURL 参数
foreach ($urls as $index => $url) {
    $ch_index = $index % $max_threads;
    curl_setopt($ch[$ch_index], CURLOPT_URL, $url);
    curl_setopt($ch[$ch_index], CURLOPT_RETURNTRANSFER, 1);
    $curlopt_array[$ch_index][] = $ch[$ch_index];
}

// 处理所有 cURL 请求
$mh = curl_multi_init();

foreach ($curlopt_array as $ch_array) {
    foreach ($ch_array as $ch) {
        curl_multi_add_handle($mh,$ch);
    }
}

// 执行多线程请求
$running = NULL;
do {
    usleep(10000);
    curl_multi_exec($mh, $running);
} while ($running > 0);

// 关闭所有 cURL
foreach ($ch as $ch) {
    curl_multi_remove_handle($mh, $ch);
    curl_close($ch);
}

// 关闭 cURL 多线程 handler
curl_multi_close($mh);

?>

Through the above code, multiple requests can be sent at the same time, thereby realizing the function of multi-threaded data capture. Then, you can use the Simple HTML DOM component to parse the page content and obtain the required data.

<?php

// 定义目标 URL 列表
$urls = array(
    'http://example.com/'
);

// 定义线程数
$max_threads = 5;

// 定义处理函数
function handle_page_data($page_data) {
    // 解析 HTML 页面
    $html = str_get_html($page_data);
    // 获取节点
    $node = $html->find('#content', 0);
    // 获取节点文本
    $text = $node->plaintext;
    // 处理数据
    // ...
}

// 创建多个 cURL 组件
$ch = array();
for ($i = 0; $i < $max_threads; $i++) {
    $ch[$i] = curl_init();
}

// 设置 cURL 参数
$curlopt_array = array();
foreach ($urls as $index => $url) {
    $ch_index = $index % $max_threads;
    curl_setopt($ch[$ch_index], CURLOPT_URL, $url);
    curl_setopt($ch[$ch_index], CURLOPT_RETURNTRANSFER, 1);
    $curlopt_array[$ch_index][] = $ch[$ch_index];
}

// 处理 cURL 请求
$mh = curl_multi_init();

foreach ($curlopt_array as $ch_array) {
    foreach ($ch_array as $ch) {
        curl_multi_add_handle($mh,$ch);
    }
}

// 处理多线程请求
$running = NULL;
do {
    usleep(10000);
    curl_multi_exec($mh, $running);

    // 处理数据
    while ($done = curl_multi_info_read($mh)) {
        $info = curl_getinfo($done['handle']);
        $page_data = curl_multi_getcontent($done['handle']);
        handle_page_data($page_data);
        curl_multi_remove_handle($mh, $done['handle']);
        curl_close($done['handle']);
    }   
    
} while ($running > 0);

// 关闭所有 cURL 组件
foreach ($ch as $ch) {
    curl_multi_remove_handle($mh, $ch);
    curl_close($ch);
}

// 关闭 cURL 多线程 handler
curl_multi_close($mh);

?>

The above code obtains the page content in a multi-threaded manner, and then processes the data of each page through the custom function handle_page_data.

Summary

This article introduces the single-threaded and multi-threaded methods of using PHP to implement crawler technology. Single-threaded is convenient and easy to use, but slower, while multi-threaded is faster, but requires Consider data interaction and processing between components. In practical applications, it is necessary to choose a suitable solution for data capture and analysis based on specific needs to maximize the value of the data.

The above is the detailed content of Basic crawler tutorial: PHP program implements single-threaded and multi-threaded data crawling. For more information, please follow other related articles on the PHP Chinese website!