node.js - nodejs大批量下载图片自动中断的问题如何解决

Question

图片列表大概有1万个图片地址，
使用nodejs下载的时候，莫名其妙就会出现错误。

基本代码如下：

var downloadImage2 = function (src) {
          var iArr = src.split("/");
        var imageName = iArr[iArr.length - 1];
        //console.log("Download begin: " + imageSrc);

        var download = function(uri, filename, callback) {
            request.head(uri, function(err, res, body) {
                // console.log('content-type:', res.headers['content-type']);
                // console.log('content-length:', res.headers['content-length']);
if (uri) {request(uri).pipe(fs.createWriteStream(filename)).on('close', callback);}
                
            });
        };

        download(src, "./images/" + imageName, function() {
            console.log('done: ' + index++);

        });
    
}

出现的错误也有点莫名其妙，有时候是：

events.js:160
      throw er; // Unhandled 'error' event
      ^

Error: Invalid URI "undefined"
    at Request.init (/Users//dirread/node_modules/request/request.js:275:31)
    at new Request (/Users/dirread/node_modules/request/request.js:129:8)
    at request (/Users/dirread/node_modules/request/index.js:55:10)
    at Request._callback (/Users//down.js:21:11)
    at self.callback (/Users//dirread/node_modules/request/request.js:187:22)
    at emitOne (events.js:96:13)
    at Request.emit (events.js:188:7)

Answer 1

Invalid URI "undefined"
Tolerate errors in src, and do not download the wrong address.

Answer 2

Try using fs.writeFile instead.

Answer 3

In fact, it is very simple to add a null check to the URI, but from my experience, the problem occurs when an error occurs when parsing the URL, resulting in some resource URLs not getting the correct address. It is recommended to investigate from this perspective.