How to use superagent and cheerio using nodejs crawler

This article mainly introduces the knowledge related to superagent and cheerio for the initial trial of nodejs crawler. It is very good and has reference value. Friends in need can refer to it

Preface

I have heard of crawlers for a long time. I started to learn nodejs in the past few days and wrote a crawler https://github.com/leichangchun/node-crawlers/tree/master/superagent_cheerio_demo to crawl the homepage of the blog park. Here is a brief summary of the article title, user name, number of reads, number of recommendations and user avatar.

Use these points:

1. The core module of node--file system

2. The third-party module used for http requests--superagent

　3. Third-party module for parsing DOM--cheerio

Please go to each link for detailed explanations and APIs of several modules. There are only simple usages in the demo.

Preparation work

Use npm to manage dependencies, and the dependency information will be stored in package.json

//安装用到的第三方模块
cnpm install --save superagent cheerio

Introduction required Function module used

//引入第三方模块，superagent用于http请求，cheerio用于解析DOM
const request = require('superagent');
const cheerio = require('cheerio');
const fs = require('fs');

Request parsing page

If you want to crawl to the content of the blog park homepage, you must first request the homepage address, get To return the HTML, superagent is used here to make http requests. The basic usage method is as follows:

request.get(url)
      .end(error,res){
      //do something     
}

Initiate a get request to the specified URL. When the request is wrong, an error will be returned (if there is no error, the error is null or undefined), res is the returned data.

After getting the html content, we need to use cheerio to parse the DOM in order to get the data we want. Cheerio must first load the target html and then parse it. The API is very similar to the jquery API. , familiar with jquery and getting started very quickly. Look directly at the code example

//目标链接 博客园首页
let targetUrl = 'https://www.cnblogs.com/';
//用来暂时保存解析到的内容和图片地址数据
let content = '';
let imgs = [];
//发起请求
request.get(targetUrl)
    .end( (error,res) => {
      if(error){ //请求出错，打印错误，返回
        console.log(error)
        return;
      }
      // cheerio需要先load html
      let $ = cheerio.load(res.text);
      //抓取需要的数据,each为cheerio提供的方法用来遍历
      $('#post_list .post_item').each( (index,element) => {
        //分析所需要的数据的DOM结构
        //通过选择器定位到目标元素，再获取到数据
        let temp = {
          '标题' : $(element).find('h3 a').text(),
          '作者' : $(element).find('.post_item_foot > a').text(),
          '阅读数' : +$(element).find('.article_view a').text().slice(3,-2),
          '推荐数' : +$(element).find('.diggnum').text()
        }
        //拼接数据
        content += JSON.stringify(temp) + '\n';
        //同样的方式获取图片地址
        if($(element).find('img.pfs').length > 0){
          imgs.push($(element).find('img.pfs').attr('src'));
        }
      });
      //存放数据
      mkdir('./content',saveContent);
      mkdir('./imgs',downloadImg);
    })

Storing data

After parsing the DOM above, the required information content has been spliced ​​and the image has been obtained URL, store it now, store the content in a txt file in the specified directory, and download the image to the specified directory

Create the directory first and use the nodejs core file system

//创建目录
function mkdir(_path,callback){
  if(fs.existsSync(_path)){
    console.log(`${_path}目录已存在`)
  }else{
    fs.mkdir(_path,(error)=>{
      if(error){
        return console.log(`创建${_path}目录失败`);
      }
      console.log(`创建${_path}目录成功`)
    })
  }
  callback(); //没有生成指定目录不会执行
}

With the specification After the directory, you can write data. The content of the txt file is already there. Just write it directly. Use writeFile()

//将文字内容存入txt文件中
function saveContent() {
  fs.writeFile('./content/content.txt',content.toString());
}

to get the link to the picture, so you need to use superagent to download the picture and save it locally. . superagent can directly return a response stream, and then cooperate with the nodejs pipeline to directly write the image content to the local

//下载爬到的图片
function downloadImg() {
  imgs.forEach((imgUrl,index) => {
    //获取图片名 
    let imgName = imgUrl.split('/').pop();

    //下载图片存放到指定目录
    let stream = fs.createWriteStream(`./imgs/${imgName}`);
    let req = request.get('https:' + imgUrl); //响应流
    req.pipe(stream);
    console.log(`开始下载图片 https:${imgUrl} --> ./imgs/${imgName}`);     
  } )
}

Effect

Execute the demo and see the effect. The data has climbed down normally

A very simple demo, it may not be that rigorous, but it is always the first small step towards node.

The above is what I compiled for everyone. I hope it will be helpful to everyone in the future.

Related articles:

Vue’s routing dynamic redirection and navigation guard examples

Vue implements uploading images to the database and displaying them to Page function example

Solve the problem of failure after using vue.js routing

The above is the detailed content of How to use superagent and cheerio using nodejs crawler. For more information, please follow other related articles on the PHP Chinese website!