PHP and Selenium: How to scrape data from different web pages?

With the development of the Internet, web crawlers have undoubtedly become an important means of obtaining information. When we face multiple different websites and need to collect some of the information, we have to focus on studying the page structure of each website. So is there a general method to crawl data on different websites? The answer is yes, this article will introduce how to use PHP and Selenium to crawl data on different web pages.

What is Selenium?

Selenium can be said to be one of the standards in the web automation testing community. It can simulate user operations in the browser, including input, clicks, drop-down box selections, file uploads, etc. We can use Selenium to implement automated testing of web pages, but here, we will use Selenium to crawl data.

Why use Selenium?

When crawling data, we often encounter some pages that can be accessed through simple curl or file_get_contents requests. However, with the development of Web technology, many websites have adopted front-end frameworks such as Ajax, Vue, and React, and the data on the page is obtained asynchronously through JavaScript. In this case, we need to simulate the operation of the browser to obtain the complete rendering result of the page. In addition, in order to prevent crawlers, some websites will use verification codes and other means for verification. At this time, we also need to use Selenium to simulate manual operations before we can continue to capture data.

Basic process of using PHP and Selenium to capture data

The first step is to install Selenium

The installation of Selenium is very simple. You only need to use Composer in the terminal to execute the following command That’s it:

composer require php-webdriver/webdriver

After the installation is completed, we need to download the driver for the corresponding browser. Taking Chrome as an example, you need to download chromedriver and put the downloaded driver in the location pointed to by the PATH variable.

Second step, start Selenium

Starting Selenium is very simple, just enter the following command in the terminal:

java -jar path/to/selenium-server-standalone-3.141.59.jar

Among them, path/to/ is Selenium’s installation path.

The third step is to use Selenium to implement data capture

With Selenium and the browser driver, we can directly use PHP to call Selenium’s API to implement automated website testing and data capture. . Let's take the example of crawling the TOP250 movie information in Douban Movies as an example:

1. Open the browser and enter the URL

use FacebookWebDriverRemoteDesiredCapabilities;
use FacebookWebDriverRemoteRemoteWebDriver;

$host = 'http://localhost:4444/wd/hub'; // Selenium本地服务地址
$capabilities = DesiredCapabilities::chrome();
$webDriver = RemoteWebDriver::create($host, $capabilities);

$url = 'https://movie.douban.com/top250';
$webDriver->get($url);

2. Click on the next page to continue Get movie information

do {
    // 获取电影列表并输出
    $list = $webDriver->findElements(WebDriverBy::cssSelector('.grid_view .item'));
    foreach ($list as $item) {
        $name = $item->findElement(WebDriverBy::cssSelector('.title'))->getText();
        $directors = $item->findElement(WebDriverBy::cssSelector('.bd p:first-child'))->getText();
        $rate = $item->findElement(WebDriverBy::cssSelector('.rating_num'))->getText();
        echo "$name $directors $rate
";
    }

    // 点击下一页
    $nextPageBtn = $webDriver->findElement(WebDriverBy::cssSelector('.paginator .next a'));
    $nextPageBtnClassName = $nextPageBtn->getAttribute('class');
    if (strpos($nextPageBtnClassName, 'disabled') === false) {
        $nextPageBtn->click();
    } else {
        break;
    }

    // 等待新页面加载完成
    $webDriver->wait()->until(WebDriverExpectedCondition::urlContains(intval($page + 1)));
} while (true);

3. Close the browser

$webDriver->quit();

With the above code, we can easily get the relevant information of the TOP250 Douban movies.

Summary

This article introduces how to use PHP and Selenium to crawl data on different websites, and explains the specific implementation steps in detail with examples. Using Selenium allows us to obtain data more easily and obtain the required information more quickly, which is very useful for some crawling tasks that require high customization.

The above is the detailed content of PHP and Selenium: How to scrape data from different web pages?. For more information, please follow other related articles on the PHP Chinese website!