This article mainly introduces the sample code of the http crawler based on node. The editor thinks it is quite good, so I will share it with you now and give it as a reference. Let’s follow the editor to take a look, I hope it can help everyone.
Every moment, whether you sleep or not, there will be massive data coming and going on the Internet, from customer service to server, and from server to server. The role completed by http's get and request is to obtain and submit data. Next, we start writing a simple small crawler to crawl the course interface of the chapter about node in the novice tutorial.
Crawl all the data on the home page of the Node.js tutorial
Create node-http.js, the code is as follows, there are detailed comments in the code, you can understand it by yourself Ha
var http=require('http');//获取http模块 var url='/nodejs/nodejs-tutorial.html';//定义node官网地址变量 http.get(url,function(res){ var html=''; // 这里将会触发data事件,不断触发不断跟新html直至完毕 res.on('data',function(data){ html +=data }) // 当数据获取完成将会触发end事件,这里将会打印初node官网的html res.on('end',function(){ console.log(html) }) }).on('error',function(){ console.log('获取node官网相关数据出错') })
In the terminal execution result, it was found that all the html of this page has been crawled down
##
G:\node\node-http> node node-http.js <!Doctype html> <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /> <meta property="qc:admins" content="465267610762567726375" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> <title>Node.js 教程 | 菜鸟教程</title> <link rel='dns-prefetch' href='//s.w.org' /> <link rel="canonical" href="http://www.php.cn/nodejs/nodejs-tutorial.html" /> <meta name="keywords" content="Node.js 教程,node,Node.js,nodejs"> <meta name="description" content="Node.js 教程 简单的说 Node.js 就是运行在服务端的 JavaScript。 Node.js 是一个基于Chrome JavaScript 运行时建立的一个平台 。 Node.js是一个事件驱动I/O服务端JavaScript环境,基于Google的V8引擎,V8引擎执行Javascript的速度非常快,性能非常好。 谁适合阅读本教程? 如果你是一个前端程序员,你不懂得像PHP、Python或Ruby等动态编程语言,.."> <link rel="shortcut icon" href="//static.runoob.com/images/favicon.ico" rel="external nofollow" rel="external nofollow" mce_href="//static.runoob.com/images/favicon.ico" rel="external nofollow" rel="external nofollow" type="image/x-icon"> <link rel="stylesheet" href="/wp-content/themes/runoob/style.css?v=1.141" rel="external nofollow" type="text/css" media="all" /> <link rel="stylesheet" href="//cdn.bootcss.com/font-awesome/4.7.0/css/font-awesome.min.css" rel="external nofollow" media="all" /> <!--[if gte IE 9]><!--> 。。。。。。。。。。 这里只展示部分不然你半天看不到头Of course, crawl it HTML is of no use to us. Now we need to do some filtering. For example, in this node tutorial, I want to know what the course catalog is, so that I can choose the ones I am interested in to learn. Let’s go directly to the code: But before that we need to download the cheerio module (cheerio is nodejs’ page crawling module, specially customized for the server, a fast, flexible and implemented jQuery core implementation. Suitable for all kinds of Web crawler program.) You can search for details on your own. The use of cheerio is very similar to the use of jquery, so you don’t have to worry about getting started.
PS G:\node\node-http> npm install cheerioCreate node-http-more.js, the code is as follows:
##
var http=require('http');//获取http模块 var cheerio=require('cheerio');//引入cheerio模块 var url='http://www.php.cn/nodejs/nodejs-tutorial.html';//定义node官网地址变量 // filer node chapter function filerNodeChapter(html){ // 将爬取得HTML装载起来 var $=cheerio.load(html); // 拿到左侧边栏的每个目录 var nodeChapter=$('#leftcolumn a'); //这里我希望我能获取的到的最终数据格式这个样子的,如此我们能知道每个目录的地址及标题 /** * [{id:,title:}] */ var chapterData=[]; nodeChapter.each(function(item){ // 获取每项的地址及标题 var id=$(this).attr('href'); var title=$(this).text(); chapterData.push({ id:id, title:title }) }) return chapterData; } //获取每个数据 function getChapterData(nodeChapter){ nodeChapter.forEach(function(item){ console.log(' 【 '+item.id+' 】'+item.title+'\n') }); } http.get(url,function(res){ var html=''; // 这里将会触发data事件,不断触发不断跟新html直至完毕 res.on('data',function(data){ html +=data }) // 当数据获取完成将会触发end事件,这里将会打印初node官网的html res.on('end',function(){ //console.log(html) // 过滤出node.js的课程目录 var nodeChapter= filerNodeChapter(html); //循环打印所获取的数据 getChapterData(nodeChapter) }) }).on('error',function(){ console.log('获取node官网相关数据出错') })
Terminal execution results and printing Out of the course catalog
G:\node\node-http> node node-http-more.js 【 /nodejs/nodejs-tutorial.html 】 Node.js 教程 【 /nodejs/nodejs-install-setup.html 】 Node.js 安装配置 【 /nodejs/nodejs-http-server.html 】 Node.js 创建第一个应用 【 nodejs-npm.html 】 NPM 使用介绍 【 nodejs-repl.html 】 Node.js REPL 【 nodejs-callback.html 】 Node.js 回调函数 【 nodejs-event-loop.html 】 Node.js 事件循环 【 nodejs-event.html 】 Node.js EventEmitter 【 nodejs-buffer.html 】 Node.js Buffer 【 nodejs-stream.html 】 Node.js Stream 【 /nodejs/nodejs-module-system.html 】 Node.js 模块系统 。。。。。。。。。。。 这里就不全部给出,你可以自己尝试着运行操作查看所有结果
Related recommendations: Detailed explanation of the web request module of Node.js crawlerNode.js Development Information Crawler Process Code SharingNodeJS Encyclopedia Crawler Example Tutorial
The above is the detailed content of Sample code sharing of http crawler under node. For more information, please follow other related articles on the PHP Chinese website!

JavaScript's application in the real world includes front-end and back-end development. 1) Display front-end applications by building a TODO list application, involving DOM operations and event processing. 2) Build RESTfulAPI through Node.js and Express to demonstrate back-end applications.

The main uses of JavaScript in web development include client interaction, form verification and asynchronous communication. 1) Dynamic content update and user interaction through DOM operations; 2) Client verification is carried out before the user submits data to improve the user experience; 3) Refreshless communication with the server is achieved through AJAX technology.

Understanding how JavaScript engine works internally is important to developers because it helps write more efficient code and understand performance bottlenecks and optimization strategies. 1) The engine's workflow includes three stages: parsing, compiling and execution; 2) During the execution process, the engine will perform dynamic optimization, such as inline cache and hidden classes; 3) Best practices include avoiding global variables, optimizing loops, using const and lets, and avoiding excessive use of closures.

Python is more suitable for beginners, with a smooth learning curve and concise syntax; JavaScript is suitable for front-end development, with a steep learning curve and flexible syntax. 1. Python syntax is intuitive and suitable for data science and back-end development. 2. JavaScript is flexible and widely used in front-end and server-side programming.

Python and JavaScript have their own advantages and disadvantages in terms of community, libraries and resources. 1) The Python community is friendly and suitable for beginners, but the front-end development resources are not as rich as JavaScript. 2) Python is powerful in data science and machine learning libraries, while JavaScript is better in front-end development libraries and frameworks. 3) Both have rich learning resources, but Python is suitable for starting with official documents, while JavaScript is better with MDNWebDocs. The choice should be based on project needs and personal interests.

The shift from C/C to JavaScript requires adapting to dynamic typing, garbage collection and asynchronous programming. 1) C/C is a statically typed language that requires manual memory management, while JavaScript is dynamically typed and garbage collection is automatically processed. 2) C/C needs to be compiled into machine code, while JavaScript is an interpreted language. 3) JavaScript introduces concepts such as closures, prototype chains and Promise, which enhances flexibility and asynchronous programming capabilities.

Different JavaScript engines have different effects when parsing and executing JavaScript code, because the implementation principles and optimization strategies of each engine differ. 1. Lexical analysis: convert source code into lexical unit. 2. Grammar analysis: Generate an abstract syntax tree. 3. Optimization and compilation: Generate machine code through the JIT compiler. 4. Execute: Run the machine code. V8 engine optimizes through instant compilation and hidden class, SpiderMonkey uses a type inference system, resulting in different performance performance on the same code.

JavaScript's applications in the real world include server-side programming, mobile application development and Internet of Things control: 1. Server-side programming is realized through Node.js, suitable for high concurrent request processing. 2. Mobile application development is carried out through ReactNative and supports cross-platform deployment. 3. Used for IoT device control through Johnny-Five library, suitable for hardware interaction.


Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

AI Hentai Generator
Generate AI Hentai for free.

Hot Article

Hot Tools

mPDF
mPDF is a PHP library that can generate PDF files from UTF-8 encoded HTML. The original author, Ian Back, wrote mPDF to output PDF files "on the fly" from his website and handle different languages. It is slower than original scripts like HTML2FPDF and produces larger files when using Unicode fonts, but supports CSS styles etc. and has a lot of enhancements. Supports almost all languages, including RTL (Arabic and Hebrew) and CJK (Chinese, Japanese and Korean). Supports nested block-level elements (such as P, DIV),

VSCode Windows 64-bit Download
A free and powerful IDE editor launched by Microsoft

EditPlus Chinese cracked version
Small size, syntax highlighting, does not support code prompt function

MantisBT
Mantis is an easy-to-deploy web-based defect tracking tool designed to aid in product defect tracking. It requires PHP, MySQL and a web server. Check out our demo and hosting services.

SublimeText3 Chinese version
Chinese version, very easy to use