


With the continuous development of tourism, tourism information has become very rich. In order to facilitate everyone to obtain more comprehensive and accurate travel information, we can use crawlers to crawl data on travel websites, analyze and process it. This article will introduce how to use PHP to crawl Ctrip travel information.
- Basics of crawlers
A crawler is an automated program that simulates users visiting a website and obtains data on the website. The crawler is generally divided into the following steps:
- Initiate a request: The crawler program will initiate an HTTP request to the target website to obtain the HTML code of the target website.
- Parse HTML: Use an appropriate parser to parse the HTML code and obtain the required data. Common parsers include regular expressions, XPath and DOM parsers.
- Storing data: Store the obtained data in a file or database for subsequent analysis and processing.
- Target website analysis
Before writing a crawler, you first need to analyze the structure and data of the target website. The target website of this crawler is the Ctrip travel website. We need to analyze the URL structure and HTML code of the website to determine how it is requested and how it is parsed.
2.1 URL structure analysis
The URL structure of Ctrip’s travel website is relatively simple. Each travel product has a unique ID number. Take browsing Beijing tourism as an example. The URL is: https://you.ctrip.com/sight/beijing1.html. Among them, beijing1 represents the ID number of Beijing Tourism.
2.2 HTML code analysis
Use the developer tools of the Chrome browser to view the HTML code of the webpage. You can find that the travel information on the page is in a div with the class "list_mod2". This div contains detailed information about each travel product, including product name, price, reviews, etc. In the crawler program, we need to use PHP's DOM parser to parse this information.
- Crawler program writing
Now that we have understood the URL structure and HTML code of the target website, we can write a crawler program to obtain travel information.
3.1 Initiate an HTTP request
First, we need to use PHP's cURL library to initiate an HTTP request to the target website and obtain the HTML code of the target website. In the code, we use a loop, starting from ID=1, to obtain the detailed information of each travel product in sequence.
//爬取ID号从1到n的旅游产品的详细信息 for($i=1;$i<=$n;$i++){ $url = "https://you.ctrip.com/sight/beijing$i.html"; //目标网站URL $ch = curl_init(); curl_setopt($ch, CURLOPT_URL, $url); curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); $html = curl_exec($ch); curl_close($ch); }
3.2 Parse HTML code
Next, we use PHP’s DOM parser to parse the HTML code of the target website and obtain the travel product information we need. In the code, we use XPath expressions to select the div with class "list_mod2" and obtain the product name, price, evaluation and other information.
$doc = new DOMDocument(); $doc->loadHTML($html); $xpath = new DOMXPath($doc); $list_mod2 = $xpath->query('//*[@class="list_mod2"]'); //选择class为"list_mod2"的div foreach($list_mod2 as $mod){ $modImg = $xpath->query('a/img/@src', $mod)->item(0)->nodeValue; //获取产品图片URL $modTitle = $xpath->query('div/h2/a', $mod)->item(0)->nodeValue; //获取产品名称 $modContent = $xpath->query('div/div/div/p/@title', $mod)->item(0)->nodeValue; //获取产品简介 $modPrice = $xpath->query('div/div/span/em', $mod)->item(0)->nodeValue; //获取产品价格 $modComment = $xpath->query('div/div/div/div/span[1]', $mod)->item(0)->nodeValue; //获取产品评价分数 $modCommentNum = $xpath->query('div/div/div/div/span[2]', $mod)->item(0)->nodeValue; //获取产品评价人数 }
3.3 Storing data
Finally, we store the obtained tourism product information in the database. In the code, we use PHP's PDO extension to connect to the database and insert the obtained product information into the database table named "product_info".
//连接数据库 $dsn = 'mysql:host=localhost;port=3306;dbname=test;charset=utf8'; $username = 'root'; $password = '123456'; $options = array(PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION); $dbh = new PDO($dsn, $username, $password, $options); //将获取到的产品信息插入到数据库 $stmt = $dbh->prepare("INSERT INTO product_info(img_url, title, content, price, comment_score, comment_num) VALUES(?, ?, ?, ?, ?, ?)"); $stmt->bindParam(1, $modImg); $stmt->bindParam(2, $modTitle); $stmt->bindParam(3, $modContent); $stmt->bindParam(4, $modPrice); $stmt->bindParam(5, $modComment); $stmt->bindParam(6, $modCommentNum); $stmt->execute();
- Summary
Using PHP to write a crawler program, you can easily grab the travel product information of Ctrip travel website and store it in the database. In actual development, we can also use multi-threading, IP proxy and other technologies to improve the efficiency and stability of the crawler program. However, it should be noted that when using crawler programs, the copyright and privacy policies of the target website should be respected to avoid infringement and abuse.
The above is the detailed content of Practical crawler practice: Use PHP to crawl Ctrip travel information. For more information, please follow other related articles on the PHP Chinese website!

php把负数转为正整数的方法:1、使用abs()函数将负数转为正数,使用intval()函数对正数取整,转为正整数,语法“intval(abs($number))”;2、利用“~”位运算符将负数取反加一,语法“~$number + 1”。

实现方法:1、使用“sleep(延迟秒数)”语句,可延迟执行函数若干秒;2、使用“time_nanosleep(延迟秒数,延迟纳秒数)”语句,可延迟执行函数若干秒和纳秒;3、使用“time_sleep_until(time()+7)”语句。

php除以100保留两位小数的方法:1、利用“/”运算符进行除法运算,语法“数值 / 100”;2、使用“number_format(除法结果, 2)”或“sprintf("%.2f",除法结果)”语句进行四舍五入的处理值,并保留两位小数。

判断方法:1、使用“strtotime("年-月-日")”语句将给定的年月日转换为时间戳格式;2、用“date("z",时间戳)+1”语句计算指定时间戳是一年的第几天。date()返回的天数是从0开始计算的,因此真实天数需要在此基础上加1。

方法:1、用“str_replace(" ","其他字符",$str)”语句,可将nbsp符替换为其他字符;2、用“preg_replace("/(\s|\ \;||\xc2\xa0)/","其他字符",$str)”语句。

php判断有没有小数点的方法:1、使用“strpos(数字字符串,'.')”语法,如果返回小数点在字符串中第一次出现的位置,则有小数点;2、使用“strrpos(数字字符串,'.')”语句,如果返回小数点在字符串中最后一次出现的位置,则有。

php字符串有下标。在PHP中,下标不仅可以应用于数组和对象,还可应用于字符串,利用字符串的下标和中括号“[]”可以访问指定索引位置的字符,并对该字符进行读写,语法“字符串名[下标值]”;字符串的下标值(索引值)只能是整数类型,起始值为0。

在php中,可以使用substr()函数来读取字符串后几个字符,只需要将该函数的第二个参数设置为负值,第三个参数省略即可;语法为“substr(字符串,-n)”,表示读取从字符串结尾处向前数第n个字符开始,直到字符串结尾的全部字符。


Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

AI Hentai Generator
Generate AI Hentai for free.

Hot Article

Hot Tools

WebStorm Mac version
Useful JavaScript development tools

MinGW - Minimalist GNU for Windows
This project is in the process of being migrated to osdn.net/projects/mingw, you can continue to follow us there. MinGW: A native Windows port of the GNU Compiler Collection (GCC), freely distributable import libraries and header files for building native Windows applications; includes extensions to the MSVC runtime to support C99 functionality. All MinGW software can run on 64-bit Windows platforms.

Dreamweaver CS6
Visual web development tools

Atom editor mac version download
The most popular open source editor

SublimeText3 English version
Recommended: Win version, supports code prompts!
