nodejs crawler framework superagent

This time I will bring you the nodejs crawler framework superagent. What are the precautions of the nodejs crawler framework superagent. Here are the actual cases, let’s take a look.

Preface

I have heard of crawlers for a long time. I started to learn nodejs in the past few days and wrote a crawlerhttps://github.com/leichangchun/node-crawlers/tree/master/superagent_cheerio_demo, crawl the article title, user name, number of reads, number of recommendations and user avatar on the homepage of the blog park, and now make a short summary.

Use these points:

1. The core module of node - File system

2. Third party used for http requests Module -- superagent

3. Third-party module for parsing DOM -- cheerio

For detailed explanations and APIs of several modules, please refer to each link. There are only simple usages in the demo.

Preparation work

Use npm to manage dependencies, and dependency information will be stored in package.json

//安装用到的第三方模块
cnpm install --save superagent cheerio

Introducing the required functional modules

//引入第三方模块，superagent用于http请求，cheerio用于解析DOM
const request = require('superagent');
const cheerio = require('cheerio');
const fs = require('fs');

Request + Parse page

If you want to climb to the content of the blog park homepage, First, you need to request the homepage address and get the returned html. Here, superagent is used to make an http request. The basic usage method is as follows:

request.get(url)
      .end(error,res){
      //do something     
}

Initiate a get request to the specified url. When the request is incorrect, an error will be returned (no In case of error, error is null or undefined), and res is the returned data.

After getting the html content, we need to use cheerio to parse the DOM in order to get the data we want. Cheerio must first load the target html and then parse it. The API is very similar to the jquery API. , familiar with jquery and getting started very quickly. Look directly at the code example

//目标链接 博客园首页
let targetUrl = 'https://www.cnblogs.com/';
//用来暂时保存解析到的内容和图片地址数据
let content = '';
let imgs = [];
//发起请求
request.get(targetUrl)
    .end( (error,res) => {
      if(error){ //请求出错，打印错误，返回
        console.log(error)
        return;
      }
      // cheerio需要先load html
      let $ = cheerio.load(res.text);
      //抓取需要的数据,each为cheerio提供的方法用来遍历
      $('#post_list .post_item').each( (index,element) => {
        //分析所需要的数据的DOM结构
        //通过选择器定位到目标元素，再获取到数据
        let temp = {
          '标题' : $(element).find('h3 a').text(),
          '作者' : $(element).find('.post_item_foot > a').text(),
          '阅读数' : +$(element).find('.article_view a').text().slice(3,-2),
          '推荐数' : +$(element).find('.diggnum').text()
        }
        //拼接数据
        content += JSON.stringify(temp) + '\n';
        //同样的方式获取图片地址
        if($(element).find('img.pfs').length > 0){
          imgs.push($(element).find('img.pfs').attr('src'));
        }
      });
      //存放数据
      mkdir('./content',saveContent);
      mkdir('./imgs',downloadImg);
    })

Storing data

After parsing the DOM above, the required information content has been spliced ​​and the image has been obtained URL, store it now, store the content in a txt file in the specified directory, and download the image to the specified directory

Create the directory first and use the nodejs core file system

//创建目录
function mkdir(_path,callback){
  if(fs.existsSync(_path)){
    console.log(`${_path}目录已存在`)
  }else{
    fs.mkdir(_path,(error)=>{
      if(error){
        return console.log(`创建${_path}目录失败`);
      }
      console.log(`创建${_path}目录成功`)
    })
  }
  callback(); //没有生成指定目录不会执行
}

With the specification After the directory, you can write data. The content of the txt file is already there. Just write it directly. Use writeFile()

//将文字内容存入txt文件中
function saveContent() {
  fs.writeFile('./content/content.txt',content.toString());
}

to get the link to the picture, so you need to use superagent to download the picture and save it locally. . superagent can directly return a response stream, and then cooperate with the nodejs pipeline to directly write the image content to the local

//下载爬到的图片
function downloadImg() {
  imgs.forEach((imgUrl,index) => {
    //获取图片名 
    let imgName = imgUrl.split('/').pop();
    //下载图片存放到指定目录
    let stream = fs.createWriteStream(`./imgs/${imgName}`);
    let req = request.get('https:' + imgUrl); //响应流
    req.pipe(stream);
    console.log(`开始下载图片 https:${imgUrl} --> ./imgs/${imgName}`);     
  } )
}

Effect

Execute the demo and see the effect. The data has climbed down normally

A very simple demo, it may not be that rigorous, but it is always the first small step towards node.

I believe you have mastered the method after reading the case in this article. For more exciting information, please pay attention to other related articles on the php Chinese website!

Recommended reading:

H5’s semantic tags

H5’s method of reading files and uploading them to the server

How to achieve the animation effect of picture rotation in html5

The above is the detailed content of nodejs crawler framework superagent. For more information, please follow other related articles on the PHP Chinese website!